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Executive  Summary 


This  Systems  Engineering  Research  Center  (SERC)  2014-2018  Technical  Plan  aligns  the  SERC  Vision  and 
Research  Strategy  with  the  Sponsor's  Core  funding  priorities.  It  describes  the  SERC  Vision,  the  Sponsor's 
needs,  and  the  SERC's  response  to  these  needs,  supported  by  research  in  Enterprises  and  Systems  of 
Systems  (ESOS),  Trusted  Systems  (TS),  Systems  Engineering  and  Systems  Management  Transformation 
(SEMT)  and  Human  Capital  Development  (HCD).  A  Grand  Challenge  statement  is  presented  for  each  of 
these  four  research  areas,  along  with  a  strategy  to  address  it.  Eleven  research  programs  have  been 
identified  to  support  this  strategy.  Research  projects  are  then  presented  which  support  each  of  these 
programs,  consisting  of  existing  and  anticipated  future  projects. 

This  2016  update  describes  progress  since  the  Assistant  Secretary  of  Defense  for  Research  and 
Engineering  approved  the  original  SERC  Technical  Plan  in  October  2013  and  annual  Core  funding  was 
appropriated  to  match  the  Technical  Plan.  In  this  update,  the  Grand  Challenges  remain  virtually 
unchanged,  with  the  focus  being  updates  to  the  research  programs  and  other  elements  in  the  plan.  In 
addition,  this  update  includes  much  greater  transition  planning  information  than  did  the  original  version. 
The  next  Technical  Plan  update  in  two  years  will  address  changes  to  the  Grand  Challenges  and  research 
strategies,  and  will  be  used  to  support  the  Department  of  Defense's  (DoD's)  5-year  comprehensive  review 
of  the  SERC. 

More  than  two-dozen  projects  have  been  executed  since  the  original  Technical  Plan  was  published,  some 
to  completion,  others  still  ongoing.  These  projects  have  been  delivering  methods,  processes,  and  tools 
(MPTs)  in  each  of  the  four  research  areas  that  define  the  SERC  research  portfolio.  Transition  has  also  been 
ongoing  and  growing,  with  many  acquisition  programs  and  defense  organizations  piloting  and  adopting 
SERC  MPTs  as  those  MPTs  have  matured.  Since  October  2013,  when  the  SERC  began  executing  this  plan, 
SERC  researchers  have  delivered  more  than  110  papers  and  technical  reports,  and  prototype  software 
implementations  of  their  methods  and  processes.  Equally  important,  SERC  collaboration  and 
infrastructure  have  grown  significantly,  as  reflected  in  the  new  SERC  Innovation  and  Demonstration  Lab, 
where  projects  can  demonstrate  their  research  both  individually  and  coordinated  across  projects. 

The  grand  challenges  the  SERC  is  addressing  are: 

ESOS  -  Create  the  foundational  SE  principles  and  develop  the  appropriate  MPTs  to  enable  the  DoD 
and  its  partners  to  model  (architect,  design,  analyze),  acguire,  evolve  (operate,  maintain,  monitor) 
and  verify  complex  enterprises  and  systems  of  systems  to  generate  affordable  and  overwhelming 
competitive  advantage  over  its  current  and  future  adversaries. 

TS  -  Achieve  much  higher  levels  of  system  trust  by  applying  the  systems  approach  to  achieving 
system  assurance  and  trust  for  the  increasingly  complex,  dynamic,  cyber-physical-human  net- 
centric  systems  and  systems  of  systems  of  the  future. 

SEMT  -  Transform  the  DoD  community's  systems  engineering  and  management  methods, 
processes,  tools  and  practices  to  enable  much  more  rapid,  concurrent,  flexible,  scalable  definition 
and  analysis  of  the  increasingly  complex,  dynamic,  multi-stakeholder,  cyber-physical-human  DoD 
systems  and  systems  of  systems  of  the  future. 
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HCD  -  Discover  how  to  dramatically  accelerate  the  professional  development  of  highly  capable 
systems  engineers  and  technical  leaders  in  DoD  and  the  defense  industrial  base  and  determine 
how  to  sustainably  implement  those  discoveries. 

Although  these  challenges  will  not  be  fully  addressed  in  the  five-year  timeline  of  this  Technical  Plan, 
progress  is  now  apparent.  For  example,  research  efforts  on  Trusted  Systems  have  yielded  a  promising 
new  way  to  protect  cyber-physical  systems  from  cyber-attacks  and  have  demonstrated  the  utility  of  that 
approach  on  unmanned  vehicles.  Research  projects  advancing  our  understanding  of  technical  leadership, 
of  how  to  accelerate  maturation  of  systems  engineers,  and  of  what  enables  systems  engineers  to  be 
effective  have  yielded  new  MPTs,  courses,  theories,  and  supporting  software  that  are  in  their  inaugural 
applications  in  government  and  industry.  Further  examples  of  progress  are  described  in  Section  5  for  each 
ongoing  project. 

The  SERC,  guided  by  this  Technical  Plan,  will  deliver  the  greatest  impact  for  DoD  and  the  Intelligence 
Community  (1C)  by: 

1.  Conducting  long-term  research  that  makes  significant  progress  on  the  grand  challenges 

2.  Transitioning  that  research  into  practice  within  DoD,  the  1C,  defense  industrial  base,  and  other 
federal  agencies;  and  by  developing  more  powerful  ways  to  facilitate  such  transition 

3.  Amplifying  sponsor  resources  by  forging  relationships  with  other  organizations  that  become 
partners,  contributing  their  resources  and  energy  to  the  SERC  and  adopting  SERC  research 

4.  Strengthening  the  existing  SERC  culture,  mechanisms  and  focus  on  collaboration 

5.  Instituting  new  approaches  to  educate  future  systems  engineers  and  engineers  that  leverage  the 
full  strength  and  diversity  of  the  collaboration 

The  strategy  described  in  this  Technical  Plan  embraces  these  principles  by  which  to  operate.  For  example, 
each  existing  Core-funded  project  has  been  receiving  an  initial  funding  level  throughout  2014  and  2015, 
which  will  be  reduced  by  approximately  20%  per  annum  through  the  end  of  2018.  That  reduction 
incentivizes  Principal  Investigators  (Pis)  to  seek  complementary  funding  from  non-Core  sources.  Core 
funds  freed  up  through  this  strategy  accumulate  in  an  investment  pool  that  funds  new  programs  and 
projects.  Besides  $14.722M  previously  spent  on  projects  that  became  Core  funded  in  2014,  the  SERC  was 
awarded  more  than  $10M  in  2008-2013  on  projects  that  completed.  Projects  that  completed  prior  to  2014 
are  not  addressed  in  this  plan. 
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1  SERC  Vision 


In  the  original  Technical  Plan  published  in  2013,  the  vision  for  the  SERC  at  the  end  of  2018  was  stated  as: 

•  IMPACT:  The  SERC  has  indeed  become  the  go-to  place  for  high-quality  SE  research  and  exploratory 
development.  Its  research  is  widely  applied  in  DoD  and  the  defense  industrial  base  with  tangible 
impact  affecting  billions  of  dollars  of  acquisitions;  its  research  results  are  woven  into  the  curricula 
of  dozens  of  university  programs  (including  many  outside  the  set  of  SERC  Collaborators)  that  are 
educating  thousands  of  students. 

•  TOP  25:  The  SERC  includes  ten  of  the  US  News  and  World  Report  (USNWR)  top-25 
Industrial/Manufacturing/Systems  Engineering  departments. 

•  STUDENTS:  SERC  Collaborators  graduate  over  half  of  the  US  MS-SE  and  PhD-SE  graduates  per  year. 
Many  PhD  graduates  join  other  SERC  universities  as  faculty  or  staff,  significantly  increasing  the 
breadth  and  depth  of  research  collaborations.  Collaborators  attract  and  educate  the  best 
students,  drawn  from  current  DoD  and  defense  industrial  base  employees  and  from  those  who 
are  attracted  to  systems  engineering  by  the  vigor  and  quality  of  Collaborator  educational 
programs. 

•  LEADERSHIP:  The  SERC  provides  much  of  the  leadership  in  SE-related  professional  societies, 
increasing  collaboration  among  them.  It  continues  to  operate  and  grow  the  Conference  on 
Systems  Engineering  Research  as  the  premier  SE  research  conference,  along  with  feeding  its 
papers  into  the  leading  SE-related  journals. 

•  KNOWLEDGE:  The  SERC  operates  the  world's  largest  and  most-visited  SE  research  website, 
including  the  largest  and  best-organized  SE  research  experience  base.  It  continues  to  provide 
leadership  in  evolving  the  SE  Body  of  Knowledge.  It  runs  the  most  widely  attended  and  highest- 
rated  SE  webcast  series. 

•  SCALE:  The  SERC  has  become  a  $20M/year  enterprise:  $5M  of  Core  funding  from  ASD(R&E);  $5M 
from  other  sponsors  in  the  DoD/IC;  $5M  to  apply  research  results  in  pilots  with  DoD  operational 
organizations;  and  $5M  in  research  and  pilots  from  outside  of  DoD.  Thus,  each  $1  of  Core  funding 
attracts  an  additional  $3  of  outside  funding. 

Over  the  past  two  years,  incremental  progress  towards  that  vision  includes: 

•  IMPACT:  Although  its  footprint  is  still  small,  SERC  research  is  being  used  in  all  Services,  in  the 
defense  industrial  base,  and  in  academia -and  that  research  use  is  steadily  growing.  For  example, 
the  Marines  are  using  SERC-developed  tools  for  tradespace  analysis  and  systems  portfolio 
analysis;  the  Navy  is  adopting  SERC  model-based  systems  engineering  techniques;  the  Air  Force  is 
applying  SERC  cost  modeling  approaches  to  manage  complex  systems;  both  the  Army  and 
Defense  Acquisition  University  are  applying  SERC  approaches  to  growing  technical  leaders;  several 
defense  contractors  are  applying  research  on  how  to  improve  the  effectiveness  of  their  systems 
engineers;  and  many  universities  have  adopted  SERC  developed  approaches  to  weave  systems 
thinking  and  systems  engineering  into  engineering  capstone  projects.  The  adoption  of  SERC 
technology  is  expected  to  grow  significantly  over  the  next  three  years,  primarily  as  a  result  of  the 
SERC's  greater  focus  on  transition  and  because  the  SERC  has  an  ever  expanding  portfolio  of 
maturing  research  available  to  early  adopters. 
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•  TOP  25:  The  SERC  Collaborator  membership  is  proving  to  be  more  stable  than  anticipated  when 
the  Technical  Plan  was  originally  written  in  2013.  Eight  SERC  Collaborators  are  in  the  top  25  in 
the  2015  USNWR  rankings  for  Industrial/Manufacturing/Systems  Engineering  programs. 

•  STUDENTS:  During  this  coming  year,  the  SERC  will  begin  to  collect  data  to  measure  progress 
against  this  aspect  of  its  vision. 

•  LEADERSHIP:  Over  the  past  two  years,  faculty  members  from  SERC  Collaborator  universities  have 
played  key  roles  within  the  International  Council  on  Systems  Engineering  (INCOSE),  the 
Accreditation  Board  for  Engineering  and  Technology  (ABET),  the  American  Institute  of  Aeronautics 
and  Astronautics  (AIAA),  and  other  professional  societies;  e.g.,  one  Stevens  professor  is  a  member 
of  the  INCOSE  Board  of  Directors,  a  Massachusetts  Institute  of  Technology  professor  become 
Editor-in-Chief  of  the  Systems  Engineering  Journal  and  a  Georgia  Tech  professor  become  the 
Editor-in-Chief  of  the  Journal  of  Enterprise  Transformation. 

•  KNOWLEDGE:  Last  year  the  SERC  launched  a  new  website  which  provides  a  much  improved 
platform  to  host  and  disseminate  important  knowledge  about  systems  engineering  and  about 
SERC  research.  In  2016  the  SERC  will  begin  offering  webcasts  on  its  research  and  related  topics. 
The  SERC  is  one  of  the  three  organizations  sponsoring  the  Systems  Engineering  Body  of 
Knowledge,  which  has  become  one  of  the  most  prominent  online  source  of  information  about 
systems  engineering  -  http://www.sebokwiki.org/  averages  almost  20,000  visitors  monthly. 

•  SCALE:  During  government  FY  2014,  SERC  awards  totaled  approximately  $10M,  including  $5M  in 
Core  funding.  Growing  total  awards  to  $20M  in  2018  will  be  challenging,  but  is  certainly  feasible. 
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2  Sponsor  Needs 


The  outlook  on  SE  needs  for  2014-2018  might  best  be  summed  up  in  this  statement  from  the  INCOSE  SE 
Vision  2025  report1:  "Large  and  complex  engineered  systems  are  key  to  addressing  [engineering]  Grand 
Challenges  and  satisfying  human  and  social  needs  that  are  physical,  psychological,  economic  and  cultural. 
However,  these  systems  must  be  embedded  in  the  prevailing  social,  physical,  cultural  and  economic 
environment,  and  the  technologies  applied  to  system  solutions  must  be  tailored  to  the  relevant  local  or 
regional  capabilities  and  resources.  Full  lifecycle  analyses  and  safe,  robust  and  sustainable 
implementation  approaches,  along  with  stable  governance  environments  are  enablers  for  successful 
system  solutions."  We  have  entered  an  era  of  great  system  complexity,  rapid  technological  change, 
growing  resource  stress,  and  societal  systems  that  are  both  enabled  and  impacted  by  advancing 
engineered  systems.  Consideration  for  the  system  as  an  enterprise  and  system  of  systems  must  be  a  core 
part  of  systems  engineering.  There  are  increasing  calls  from  all  systems  stakeholders  for  foundational 
tools  that  reflect  the  human,  organizational,  and  societal  aspects  of  engineered  systems.  These 
foundations  must  inform  future  research  in  systems  engineering.  MPTs  that  consider  the  system  in  use, 
and  as  a  construct  of  a  large  development  enterprise,  are  needed  to  strengthen  future  decision-making. 
The  underlying  challenges  of  our  sponsor  base  rest  on  critical  infrastructures  that  must  be  efficient, 
resilient,  safe  and  secure,  for  which  there  is  an  increasing  movement  toward  cyber-physical 
infrastructures  and  machine  automation,  with  lifecycle  considerations  of  sustainable  resource  use. 

These  foundations  are  pervasive  in  the  needs  of  our  DoD  and  1C  sponsors.  On  April  9,  2015, 
Undersecretary  of  Defense  for  Acquisition,  Technology  &  Logistics  (USD  (AT&L))  Frank  Kendall  introduced 
the  third  in  a  series  of  DoD  Better  Buying  Power  initiatives  (BBP).  BBP3.0,  "Achieving  Dominant 
Capabilities  through  Technical  Excellence  and  Innovation,"  places  a  renewed  emphasis  on  the 
effectiveness  of  DoD  research  and  development  activities  including  science  and  technology,  system 
development,  experimentation  and  prototyping,  full-scale  development,  and  technology  insertion  into 
fielded  products2.  The  needs  for  new  and  evolving  SE  MPTs  are  reflected  in  a  number  of  BBP3.0  initiatives, 
highlighted  as  follows: 

•  Improve  decision  makers'  ability  to  understand  and  mitigate  technical  risk 

•  Increase  the  use  of  prototyping  and  experimentation 

•  Emphasize  technology  insertion  and  refresh  in  program  planning 

•  Use  Modular  Open  Systems  Architecture  to  stimulate  innovation 

•  Strengthen  Cybersecurity  throughout  the  program  lifecycle 

•  Anticipate  and  plan  for  responsive  and  emerging  threats  by  building  stronger  partnerships  of 
acquisition,  requirements,  and  intelligence  communities 

•  Reduce  cycle  times  while  ensuring  sound  investments 

•  Improve  the  return  on  investment  in  DoD  laboratories 

•  Increase  the  productivity  of  Independent  Research  and  Development  and  Corporate  Research  and 
Development 

•  Strengthen  organic  engineering  capabilities 

•  Increase  DoD  support  for  Science,  Technology,  Engineering,  and  Math  (STEM)  education 


1  See  http://www.incose.org/docs/default-source/aboutse/se-vision-2025.pdf?sfvrsn=4. 

2  See  http://bbp.dau.mil. 
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BBP3.0  reflects  a  call  for  much  stronger  integration  of  technology  and  its  use,  better  tradespace 
understanding  and  decision-making,  modular  architectures  and  much  better  knowledge  management, 
and  consideration  of  cyber  and  physical  interaction.  A  primary  BBP3.0  research  area  of  focus  is  supported 
by  the  DoD  Engineered  Resilient  Systems  (ERS)  science  and  technology  program3  and  by  related  SERC 
SEMT  research.  ERS  key  desired  capabilities  include  tradespace  tools  and  analytics  for  improved  decision 
making,  assessment  of  technical  and  program  options,  requirements  analysis  and  generation,  rapid 
generation  of  alternatives,  as  well  as  virtual  prototyping  and  experimentation.  ERS  also  includes 
development  requirements  for  collaborative  knowledge  management  and  government  managed 
modeling  and  simulation  capabilities.  Longer-range  capabilities  include  systems  architectures  and 
concepts  of  operations  (CONOPs),  as  well  as  lifecycle  modeling  and  intellectual  property  management. 

System  Security  Engineering  remains  a  further  strong  area  of  focus  for  software-intensive  system 
development,  cyber-physical  capabilities,  and  to  address  the  need  for  much  higher  levels  of  system  trust 
that  are  required  as  systems  become  increasingly  capable  and  critical.  System  security  engineering  and 
risk-based  frameworks  for  trust  have  become  a  core  SE  need  for  every  system,  whether  open  or  restricted. 
The  risk-based  approach  is  evidenced  with  a  major  sponsor  shift  to  the  National  Institute  of  Standards  and 
Technology's  (NIST's)  Risk  Management  Framework  requirements  for  government  systems4 5,  and  will  lead 
to  a  need  for  research  in  full  lifecycle  approaches  that  increase  system  trust  in  design,  reflect  the  human 
elements  of  the  system,  and  provide  sustainable  approaches  reflecting  emergent  properties  of  the  system 
as  a  whole.  The  DoD  released  its  new  cyber  strategy  in  2015s,  recognizing  both  our  commitment  to  "an 
open,  secure,  interoperable,  and  reliable  internet"  and  that  "the  increased  use  of  cyberattacks  as  a 
political  instrument  reflects  a  dangerous  trend  in  international  relations."  Our  DoD  and  1C  sponsors  are 
critically  dependent  on  the  successful  defense  of  both  DoD  networks  and  systems  and  the  US  critical 
infrastructure  from  cyberattacks.  This  can  be  captured  in  an  enterprise  view  reflecting  information 
sharing  and  coordination,  public  and  private  integration  of  security  and  trust,  and  global  alliances. 
Emerging  SE  challenges  include  recognition  of  the  cyber  mission  in  system  CONOPs,  convergence  of 
enterprise  and  mission  architecture  frameworks,  integration  of  human  cyber  operators  in  system 
requirements,  technical  and  enterprise  risk  management  approaches,  resilient  system-of-systems 
architectures  for  information  and  operations,  and  rapid  integration  of  commercial  innovation  into  defense 
systems. 

As  captured  through  DoD's  Reliance  21  framework  and  related  Service  and  other  activities6,  reliable 
autonomy  will  be  enabled  in  part  by  SE  tools  that  balance  human  oversight  of  autonomous  systems  with 
appropriate  autonomy  and  man-machine  interaction.  Such  tools  are  needed  to  help  autonomous  systems 
safely  accomplish  complex  tasks  in  all  environments.  This  includes  new  MPTs  for  test  and  evaluation  of 
autonomous  behaviors  and  decisions.  Human  interaction  that  is  intelligent,  adaptive,  and  intuitive  will 
require  much  greater  integration  of  MPTs  that  blend  human  and  machine  workflows.  This  will  support 
man-machine  teaming  but  will  also  benefit  all  engineered  systems.  SE  MPTs  that  support  massive  data 
architectures  and  alignment  of  data  intensive  algorithms  with  appropriate  CONOPs  must  also  be 
developed. 


3  See  http://www.dtic.mil/ndia/2014svstem/16903WedTrack2Holland.pdf. 

4  See  http://csrc.nist.gov/publications/nistpubs/800-37-revl/sp80Q-37-revl-final.pdf. 

5  See  http://www.defense.gov/News/Special-Reports/0415  Cyber-Strategy. 

6  See  http://www.acq.osd.mil/chieftechnologist/reliance21.html. 
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The  complexities  of  future  systems  will  only  increase  the  need  for  development  of  complex  SE  skills  and 
knowledge,  and  the  critical  role  of  leadership  and  management  skills  at  the  core  of  the  systems  disciplines. 
INCOSE's  SE  Vision  2025  lists  the  following  core  systems  engineering  skills  as  key  to  future  SE:  domain 
specific  application  and  technical  knowledge,  full  lifecycle  system  experience,  systems  engineering 
foundations,  systems  and  specialty  engineering  methods,  technical  leadership,  socio-technical 
competency,  and  software  based  tools.  SE  Vision  2025  also  notes  that  systems  engineering  skills  are  not 
just  for  systems  engineers,  but  rather  they  are  important  for  all  engineers.  In  addition,  all  systems  decision 
makers  should  have  experience  in  systems  thinking.  Strong  core  STEM  education  and  systems  thinking 
introduced  early  in  schools  are  needed  to  build  the  foundation  of  future  systems  engineers.  This  starts 
with  stronger  organic  skills  in  systems  and  other  engineering  foundations.  SE  education  must  be 
complemented  by  effective  leadership  and  management  development.  SE  is  a  lifelong  learning  process 
that  builds  from  technical  depth,  breadth  of  experience  and  knowledge,  and  strong  leadership  and 
communication.  SE  research  will  develop  the  tools  for  managing  complexity,  but  also  must  develop  the 
education  programs  that  support  effective  analyses  and  decision  making  of  those  who  will  use  them. 
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3  SERC  Response  To  Sponsor  Needs 


The  SERC  actively  manages  its  research  portfolio,  looking  for  and  nurturing  synergies  between  projects. 
The  SERC  works  with  its  sponsor  to  identify  projects  that  can  have  greater  impact  on  DoD's  strategic  SE 
research  needs.  One  such  approach  is  the  New  Project  Incubator,  described  in  Section  6,  in  which  SERC 
Collaborators  propose  new  research  ideas,  with  the  most  promising  projects  being  given  limited  funds  to 
support  their  early  development. 

Long-term  project  funding  has  been  especially  evident  since  2012,  when  the  majority  of  new  funding 
began  being  spent  on  multi-year  higher-impact  projects.  Most  projects  are  now  being  conceived  and 
proposed  as  multi-phase,  multi-year  efforts;  for  example,  the  Experience  Accelerator  Project,  which  is 
attempting  to  develop  ways  to  greatly  reduce  the  time  needed  to  mature  an  effective  systems  engineer, 
is  being  executed  as  a  5-year  project  to  deliver  a  strong  foundational  capability,  validate  it,  and  transition 
it  to  early  adopters.  Additional  sponsors  and  funding  are  being  sought  to  continue  growing  that  capability 
and  to  deliver  even  greater  value,  consistent  with  the  SCALE  element  of  the  SERC  vision  described  earlier 
in  Section  1. 

In  coordination  with  its  sponsors,  the  SERC  has  structured  its  research  portfolio  into  four  thematic  areas, 
as  shown  in  Figure  3-1  and  described  further  below: 


Figure  3-1.  The  Four  Thematic  Areas  Being  Addressed  by  SERC  Research  Tasks  and  Priorities 
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•  Enterprises  and  Systems  of  Systems:  Providing  ways  to  develop,  characterize  and  evolve  very 
large-scale  systems  composed  of  smaller  systems,  which  may  be  technical,  socio-technical,  or  even 
natural  systems.  These  are  complex  systems  in  which  the  human  behavioral  aspects  are  often 
critical,  boundaries  are  often  fuzzy,  interdependencies  are  dynamic,  and  emergent  behavior  is  the 
norm.  Research  must  enable  prediction,  conception,  design,  integration,  verification,  evolution, 
and  management  of  such  complex  systems. 

•  Trusted  Systems:  Providing  ways  to  conceive,  develop,  deploy  and  sustain  systems  that  are  safe, 
secure,  dependable  and  survivable.  Research  must  enable  prediction,  conception,  design, 
integration,  verification,  evolution  and  management  of  these  emergent  properties  of  the  system 
as  a  whole,  recognizing  these  are  not  just  properties  of  the  individual  components  and  that  it  is 
essential  that  the  human  element  be  considered. 

•  System  Engineering  and  System  Management  Transformation:  Providing  ways  to  acquire 
complex  systems  with  rapidly  changing  requirements  and  technology,  which  are  being  deployed 
into  evolving  legacy  environments.  Decision-making  capabilities  to  manage  these  systems  are 
critical  in  order  to  determine  how  and  when  to  apply  different  strategies  and  approaches,  and 
how  enduring  architectures  may  be  used  to  allow  an  agile  response.  Research  must  leverage  the 
capabilities  of  computation,  visualization,  and  communication  so  that  systems  engineering  and 
management  can  respond  quickly  and  agilely  to  the  characteristics  of  these  new  systems  and  their 
acquisitions. 

•  Human  Capital  Development:  Providing  ways  to  ensure  that  the  quality  and  quantity  of  systems 
engineers  and  technical  leaders  provide  a  competitive  advantage  for  the  DoD  and  defense 
industrial  base.  Research  must  determine  the  critical  knowledge  and  skills  that  the  DoD  and  1C 
workforce  require  as  well  as  determine  the  best  means  to  continually  impart  that  knowledge  and 
skills. 

These  four  thematic  areas  are  further  divided  into  eleven  programs  described  below.  These  programs 
have  the  potential  to  make  a  transformative  impact  on  DoD  and  the  1C.  The  SERC  Research  Council7,  which 
includes  some  of  the  most  capable  researchers  in  the  field,  continues  to  help  shape  this  portfolio. 

•  Enterprises  and  Systems  of  Systems 

Enterprise  Modeling:  Create,  validate,  and  transition  MPTs  to  model  the  socio-technical 
aspects  of  complex  systems  of  system  and  enterprise  systems,  including  developing  and 
populating  a  framework  to  integrate  models  created  using  diverse  methods  and  tools 

System  of  Systems  Modeling  and  Analysis:  Create,  validate,  and  transition  MPTs  for  analyzing 
and  evolving  systems  of  systems  and  provide  support  for  their  technical  assessment,  including 
through  a  "workbench"  of  analytic  tools 

•  Trusted  Systems 

Systemic  Security:  Create,  validate,  and  transition  MPTs  to  ensure  systemic  security  using 
knowledge  of  system  objectives  and  operation 

Systemic  Assurance:  Create,  validate,  and  transition  MPTs  to  provide  systemic  assurance  of 
safety,  reliability,  availability,  maintainability,  evolvability,  and  adaptability 


7  See  http://www.sercuarc.org/serc-research-council/. 
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•  Systems  Engineering  and  Systems  Management  Transformation 

Affordability  and  Value  in  Systems:  Create,  validate,  and  transition  MPTs  to  make  better 
decisions  on  affordability  and  value  in  systems 

Quantitative  Risk:  Create,  validate,  and  transition  MPTs  to  improve  risk  identification,  analysis 
tracking  and  management  in  acquisition  and  sustainment  programs 

Interactive  Model-Centric  Systems  Engineering  (IMCSE):  Create,  validate,  and  transition  MPTs 
to  rapidly  model  the  critical  aspects  of  systems,  especially  those  that  facilitate  collaborative 
system  development 

Agile  Systems  Engineering:  Create,  validate,  and  transition  MPTs  that  enable  rapid,  flexible 
and  adaptive  SE  that  can  be  applied  for  many  kinds  of  systems  in  many  types  of  development 
contexts 

•  Human  Capital  Development 

Evolving  Body  of  Knowledge:  Establish  active  communities  and  mechanisms  that  create  and 
support  living  bodies  of  SE  knowledge 

Experience  Acceleration:  Develop  an  open  source  community  that  creates,  validates,  and 
transitions  technology  and  content  for  the  use  of  experiential  technology  to  educate  systems 
engineers  and  technical  leaders 

SE  and  Technical  Leadership  Education:  Create,  validate,  and  transition  curricula  and  practices 
to  support  the  instruction  and  learning  of  systems  and  technical  leadership  for  inexperienced 
students  in  college  and  experienced  professionals 

Between  October  1,  2013  and  November  30,  2015,  research  on  these  eleven  programs  has  been  packaged 
into  30  projects  which  have  been  awarded  more  than  $10M  in  Core  funds  plus  more  than  $5M  from  other 
DoD  organizations,  including  all  the  Services,  Defense  Acquisition  University,  and  elements  of  the 
Intelligence  Community.  In  several  cases,  those  non-Core  funds  augmented  existing  projects;  e.g., 
substantial  support  for  the  Helix  project,  begun  under  Core  funding  as  part  of  the  Evolving  Body  of 
Knowledge  program,  has  come  from  the  Intelligence  Community.  In  many  cases,  the  SERC  launched  new 
research  projects  as  part  of  existing  programs;  e.g.,  the  Army  funded  a  project  on  Technical  Leadership 
Development  as  part  of  the  SE  and  Technical  Leadership  Education  program,  and  the  Marines  have  funded 
several  efforts  to  enhance  the  Framework  for  Assessing  Cost  and  Technology  (FACT),  which  is  part  of  the 
Affordability  and  Value  in  Systems  program.  All  of  these  projects  contribute  towards  achieving  the  Grand 
Challenges  described  earlier  in  the  Executive  Summary. 
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4  Objectives,  Principles,  Approach  and  Transition  Planning 


4.1  Objectives 

The  SERC  will  have  the  greatest  impact  on  DoD  and  the  1C  by: 

1.  Conducting  long-term  research  that  makes  significant  progress  on  the  grand  challenges 

2.  Transitioning  that  research  into  practice  within  DoD,  the  1C,  defense  industrial  base,  and  other 
federal  agencies;  and  developing  more  powerful  ways  to  facilitate  such  transition 

3.  Amplifying  sponsor  resources  by  forging  relationships  with  other  organizations  that  become 
partners,  contributing  their  resources  and  energy  to  the  SERC  and  adopting  SERC  research 

4.  Strengthening  the  existing  SERC  culture,  mechanisms  and  focus  on  collaboration 

5.  Instituting  new  approaches  to  educate  future  systems  engineers  and  engineers  that  leverage  the 
full  strength  and  diversity  of  the  collaboration 

These  approaches  align  with  the  SERC's  four  Operational  Principles: 

1.  Conduct  innovative,  high-impact  research 

a.  Focus  on  research  efforts  that  have  the  potential  of  increasing  the  security  and  prosperity 
of  the  nation 

b.  Focus  on  research  which  addresses  future  systems  needs 

c.  Focus  research  efforts  on  systems  which  can  be  generalized  beyond  a  given  domain  and 
transform  the  discipline 

d.  Only  perform  tasks  which  are  research  oriented  (usually  publishable  when  not  classified 
or  otherwise  restricted) 

2.  Translate  proof-of-principle  prototypes  to  impactful  applications 

Work  to  ensure  that  there  is  a  path  from  research  results  to  impact  for  the  security  and 
prosperity  of  the  nation 

3.  Strengthen  and  leverage  the  research  network 

a.  Ensure  that  the  research  is  conducted  by  the  best  available  resources 

b.  Bring  in  new  Collaborators  and  partnering  organizations  and  institutions  who  provide 
long-term  strategic  benefit 

c.  Focus  on  creating  a  network  of  academics,  industry  and  government  that  is  sustainable 

4.  Prepare  the  next  generation 

Provide  a  focus  on  education  and  training  research,  both  in  research  on  education  and 
training,  and  in  the  actual  education  and  training  of  researchers,  graduate  students,  and 
practitioners 


4.2  Approach 

The  general  approach  to  creating  the  original  Technical  Plan  was  first  to  identify  Grand  Challenges  in  each 
of  the  four  SERC  thematic  research  areas  shown  in  Figure  3-1.  These  Grand  Challenges  were  formulated 

Contract  Number:  HQ0034-1 3-D-0004  February  18,  2016  SERC  201 4-2018  Technical  Plan  Update 


9 


UNCLASSIFIED  /APPROVED  FOR  PUBLIC  RELEASE 


to  provide  a  point  of  integration  between  existing  programs  in  each  research  focus  area,  and  also  to 
provide  opportunities  to  generate  new,  related  research  areas.  These  Grand  Challenges  also  provide 
inspiration  and  an  integration  point  for  non-SERC  universities,  federally  funded  research  and  development 
centers  (FFRDCs),  other  University  Affiliated  Research  Centers  (UARCs)  and  industry  researchers  to 
perform  collaborative  research  and  provide  natural  transition  into  use.  SERC  management  worked  with 
the  Research  Council8,  Principal  Investigators  and  others  to  craft  the  Grand  Challenges,  objectives,  and 
strategy  for  each  of  the  four  research  focus  areas,  and  to  lay  out  program  descriptions,  timelines, 
anticipated  results,  and  resources  required. 

Additionally,  this  Technical  Plan  continues  to  assume  that: 

1.  Researchers  will  be  incentivized  to  find  some  of  their  resources  from  outside  Core  funds.  This 
could  come  in  the  form  of  matching  funds  or  other  forms  of  resources. 

2.  Researchers  will  be  incentivized  to  transition  their  results  into  practice.  Each  project  will  have  a 
transition  plan  in  place  when  the  project  begins  with  the  opportunity  for  additional  downstream 
funding  to  facilitate  transition  to  practice  and  to  develop  educational  materials  and  courses  based 
on  research  results  that  will  be  shared  by  all  SERC  collaborating  institutions. 

3.  Seed  funding  will  be  available  to  explore  novel  and  promising  ideas  that  may  be  the  sources  of 
future  breakthroughs.  Through  an  open  solicitation  process  with  all  of  the  SERC  Collaborators, 
these  ideas  will  be  selected  by  the  sponsors,  SERC  Research  Council  and  SERC  leadership.  This  past 
year,  the  SERC  began  to  solicit,  select  and  provide  that  seed  funding  for  the  first  time.  An  open 
call  to  all  Collaborators  led  to  more  than  two-dozen  proposals,  of  which  the  five  most  promising 
were  funded  as  incubator  tasks.  Some  may  receive  additional  Core  funding  beginning  in  early 
2016  or  become  the  basis  to  solicit  sponsorship  outside  Core  funding. 

4.  Shared  IT  infrastructure  will  be  available  for  use  by  every  research  project. 


4.3  Transition  Planning 

Research  in  systems  engineering  is  atypical.  Traditionally,  research  discovers  new  ideas,  new  properties, 
or  new  relationships,  leaving  it  to  engineers  to  take  these  ideas  and  make  them  useful.  Systems 
engineering  research  usually  involves  both  the  early  discovery  and  their  packaging  for  useful  application. 
The  value  of  systems  engineering  research  is  in  ensuring  that  other  systems  engineers  can  more  effectively 
create  value  for  their  stakeholders.  No  matter  what  insights  SERC  researchers  achieve  through  their 
research,  until  they  are  validated  by  practicing  engineers  and  shown  to  be  useful  in  effective  development 
and  evolution  of  safe,  reliable,  and  useful  systems,  they  are,  per  M.  Poincare,  "useless  contraptions."  It  is 
for  this  reason  that  the  SERC  includes  transition  as  an  increasingly  important  part  of  its  research 
methodology  and  focus. 

The  SERC  approaches  transition  in  a  number  of  ways,  beginning  when  the  research  effort  is  defined. 
Research  plans  specify  a  variety  of  transition  actions.  The  goal  is  to  get  "useful  combinations"  of  SE  MPTs 
into  the  hands  of  SERC  sponsors  and  stakeholders  as  quickly  and  efficiently  as  possible.  MPTs  are  the 
SERC's  technology.  Effective  transition  into  application  is  key  to  providing  real  systems  engineering 
research  value. 


8  See  http://www.sercuarc.org/serc-research-council. 
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As  shown  in  Figure  4-1,  many  different  customer  motivations  affect  their  readiness  to  adopt  new 
technology.  The  initial  target  for  SERC  MPTs  is  the  innovators  and  early  adopters.  A  SERC  MPT  successfully 
transitioned  to  innovators  and  early  adopters  would  be: 

•  Applied  by  a  small  number  of  practitioners,  generally  with  substantial  assistance  from  the 
research  team 

•  Demonstrably  and  credibly  delivering  its  intended  value  to  early  adopters 

•  Taught  in  university  programs  associated  with  the  research  team 

•  Published  in  several  articles  and  conferences 

•  Sustained  largely  by  SERC  resources  and  infrastructure  with  some  support  from  elsewhere  that 
has  the  potential  to  scale  up  the  ability  for  adoption 

However,  major  impact  is  realized  when  the  MPTs  are  transitioned  to  the  early  majority.  A  SERC  MPT 
successfully  transitioned  to  the  early  majority  would  be: 

•  Widely  applied  within  its  potential  market  of  practitioners 

•  Demonstrably  and  credibly  delivering  its  intended  value  when  applied 

•  Widely  taught  in  relevant  university  programs 

•  Articulated  in  books,  videos,  papers,  social  media,  and  other  knowledge  channels 

•  Sustained  and  improved  largely  by  resources  and  infrastructure  outside  the  SERC,  including  having 
commercial  quality  tooling,  training,  and  a  cadre  of  experts  that  aid  in  its  application 

Once  research  has  been  successfully  transitioned  to  the  early  majority,  market  and  environmental  forces 
are  usually  sufficient  to  complete  the  transition  to  the  late  majority  and  laggards  who  are  usually 
convinced  by  the  results  achieved  by  the  earlier  adopters  to  satisfy  their  important  needs. 


Want  to  make 
discontinuous 
changes  in 


Want  to  make 
improvements  in 
their  business 


new  unprover 
technology  - 
want  to  be  firs 


Passionate  to 


Figure  4-1.  Classification  of  Technology  Adopters9 


9 


From  "Crossing  the  Chasm",  3rd  Edition  by  Geoffrey  Moore,  2014. 
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As  the  SERC  has  continued  to  grow  and  mature  over  the  past  seven  years,  the  organization  has  gained 
significant  experience  in  the  area  of  transition,  learning  important  lessons  on  what  is  and  is  not  effective. 
In  addition,  the  SERC  has  proactively  formed  partnerships  to  strengthen  the  transition  pipeline,  building 
an  active  network  of  systems  researchers  and  practitioners.  Strong  relationships  have  been  forged  with 
several  professional  organizations,  including  INCOSE  and  the  National  Defense  Industrial  Association 
(NDIA)  Systems  Engineering  Division.  However,  as  a  research  center,  the  SERC  has  inherent  limitations  in 
the  scale  at  which  it  can  directly  support  transition.  Therefore,  the  SERC  will  generally  enable  and  directly 
support  transition  only  to  a  small  number  of  innovators  and  early  adopters.  At  their  discretion,  SERC 
Collaborators  may  seek  to  scale  MPT  transition  to  a  large  group  of  innovators  or  early  adopters  or  even 
seek  broader  transition  of  an  MPT.  Generally,  the  SERC  will  play  only  a  very  limited  or  no  role  in  that  larger 
transition.  The  universities  that  make  up  the  SERC  may  take  on  this  role  outside  of  the  SERC  contract. 

Based  on  past  experiences,  six  principles  have  emerged  that  underlie  effective  transition  readiness  and 
progress  as  shown  in  Table  4-1.  These  principles  have  just  recently  been  documented,  but  have  been 
applied  in  varying  degrees  since  the  SERC  was  founded  in  2008. 


Table  4-1.  Six  Principles  of  Successful  Transition 


Name 

Principle 

Plan  Early 

Make  successful  transition  an  explicit  and  well-planned  goal  from  the  project's  outset 

Balance  Long  and  Short 

Term 

Balance  the  desire  for  longer-term  higher  impact  research  with  the  importance  of 
shorter-term  utility,  incrementally  delivering  results 

Pilot  Continuously 

Continuously  engage  both  practitioner  and  student  pilot  groups  to  improve  the  utility 
and  confirm  the  validity  of  the  research 

Engage  Community 

Build  strong  engagement  with  outside  communities  who  can  become  advocates  and 
adopters 

Support  Centrally 

Strengthen  SERC-wide  infrastructure  and  incentives  to  help  projects  successfully 
transition  their  research 

Productize 

As  adoption  scale  grows,  create  mature  tools,  guides,  and  other  artifacts  to  help 
adopters  succeed,  relying  where  appropriate  on  outside  organizations  that  will 
mature  research-grade  MPTs  into  production-quality  products  and  services 

As  noted,  each  research  project  needs  to  establish  a  transition  plan  based  on  the  principles  described 
above.  Once  this  has  been  completed,  the  transition  readiness  of  the  MPTs  resulting  from  the  research 
needs  to  be  characterized.  Two  dimensions  characterize  the  readiness  of  the  MPTs  for  transition: 
relevance  and  practicality.  Relevance  is  determined  by  the  ability  of  the  new  MPT  to  help  Innovators  and 
Early  Adopters  perform  a  valuable  activity  better  than  they  otherwise  reasonably  could;  e.g.,  does  a  new 
approach  to  understanding  the  "ilities"  of  a  system  architecture  really  offer  relevant  insights  on  reliability, 
safety,  etc.  that  other  MPTs  that  analyze  architectures  do  not?  An  MPT  has  high  relevance  when  it  has 
intrinsically  high  value  and/or  differentiating  capabilities;  e.g.,  being  able  to  predict  with  high  confidence 
the  cost  of  building  a  system  of  interest  or  being  able  to  develop  an  accurate  model  of  the  behavior  of  a 
system  in  half  the  time  it  would  take  using  other  available  MPTs. 

Practicality  is  determined  by  how  easily  Innovators  and  Early  Adopters  can  cost-effectively  apply  it;  e.g., 
is  data  required  for  the  MPT  reasonably  available,  is  automation  available  to  perform  the  MPT  activities, 
does  the  MPT  work  on  "real"  problems?  The  bar  of  acceptability  for  both  relevance  and  practicality  is 
raised  when  the  MPTs  are  being  transitioned  to  Early  Adopters  rather  than  to  Innovators.  An  MPT  has 
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high  practicality  when  practitioners  who  are  skilled  in  the  activity,  but  not  originally  skilled  in  the  MPT, 
can  cost-effectively  learn  it  and  consistently  and  cost-effectively  apply  it  to  produce  valuable  results. 

Once  transition  readiness  has  been  characterized  for  a  project  or  program,  corrective  actions  or 
improvements  can  be  made  based  on  the  transition  principles  described  earlier.  However,  it  is  important 
to  measure  the  transition  progress  of  the  MPT  to  determine  the  effectiveness  of  these  measures.  Two 
dimensions  can  characterize  how  much  a  SERC  MPT  has  transitioned  to  Innovators  and  Early  Adopters: 
approval  and  adoption.  Approval  is  determined  by  how  much  better  adopting  practitioners  believe  the 
MPT  succeeds  at  delivering  value  relative  to  alternative  MPTs.  It  is  the  driving  force  for  adoption.  An  MPT 
has  high  approval  when  practitioners  routinely  praise  the  MPT's  impact,  cite  evidence  of  that  impact,  and 
advocate  for  its  adoption.  Adoption  is  a  measure  of  how  widely  the  MPT  is  used  by  practitioners  relative 
to  the  potential  market  of  the  MPT.  An  MPT  has  high  adoption  when  practitioners  from  many  diverse 
organizations  use  the  MPT,  it  is  widely  taught  in  universities,  and  descriptions  of  it  are  available  from  many 
sources. 

Finally,  one  of  the  objectives  of  this  Technical  Plan  is  to  help  the  SERC  maintain  a  healthy  diverse  research 
portfolio  that  supports  a  steady  pipeline  of  transitioning  MPTs.  As  such,  each  research  project  will  have 
a  transition  plan  in  place  based  on  a  stated  set  of  actions  supported  by  the  six  transition  principles.  In 
addition,  each  research  project  will  have  its  transition  state  characterized  based  on  transition  readiness 
{relevance  and  practicality )  and  its  transition  progress  ( approval  and  adoption )  based  on  project  evidence. 
This  information  will  provide  the  SERC  and  its  sponsor  the  ability  to  determine  the  appropriate  mix  of 
transition  characteristics  to  support  their  strategic  objectives,  to  take  action  when  necessary  and  to 
provide  researchers  with  the  tools  to  improve  their  transition  effectiveness. 
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5  Focus  Areas,  Programs,  and  Projects 


Since  October  2013,  every  project  in  the  SERC  research  portfolio  has  fit  into  one  of  eleven  programs  in 
the  four  research  areas  shown  in  Figure  3-1.  This  Technical  Plan  primarily  describes  the  allocation  of  Core 
funds  (approximately  $5M  annually)  to  13  existing  projects  (shown  in  Table  5-1)  and  the  potential 
allocation  to  new,  yet  unidentified,  projects.  However,  the  SERC  research  portfolio  is  much  larger  and 
more  diverse  than  would  be  possible  with  just  Core  funding.  Between  October  1,  2013  and  November  30, 
2015  more  than  $5M  has  been  awarded  to  SERC  projects  funded  by  the  Services,  Defense  Acquisition 
University  (DAU),  the  1C,  and  other  organizations  in  DoD. 

Besides  directly  funding  SERC  projects,  sponsors  may  provide  coordinated  funding  or  in-kind  resources 
that  contribute  to  the  execution  of  SERC  projects;  e.g.,  MITRE  has  coordinated  some  of  its  research  efforts 
with  tasks  in  the  Systems-Aware  Security  Project. 

Sometimes  non-Core  funds  augment  previously  existing  Core-funded  projects  e.g.,  funding  contributed 
by  the  Intelligence  Community  towards  the  Helix  Project.  At  other  times,  non-Core  funds  support  new 
projects  to  which  Core  funds  are  later  added;  e.g.,  DAU  initially  funded  the  Experience  Accelerator  Project 
and  continues  to  do  so,  but  Core  funds  now  also  contribute.  Finally,  there  have  been  a  number  of  projects 
spawned  by  non-Core  investment  which  have  no  corresponding  Core  funding  (such  as  a  Navy  project 
exploring  how  to  adopt  model-based  systems  engineering),  but  which  help  address  one  or  more  Grand 
Challenges. 
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Figure  5-1.  Core  Funding  Distribution  across  Research  Focus  Areas 


Figure  5-1  shows  the  relative  distribution  of  Core  funding  between  the  four  research  focus  areas  over  the 
five  years  of  the  Plan  plus  the  funding  before  the  Plan  started.  These  funding  levels  are  approximate, 
intended  to  serve  as  the  basis  for  program  and  project  prioritization  and  sizing  decisions.  Awarded  funds 
may  be  expended  across  fiscal  years. 
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The  general  philosophy  is  that  each  new  project  receives  steady  funding  for  two  years,  giving  that  project 
time  to  establish  its  research  approach  and  begin  to  obtain  early  results.  Funding  for  that  project  is  then 
reduced  by  20%  annually  to  incentivize  the  PI  to  find  some  funding  from  non-Core  sources.  In  some  cases, 
projects  will  end  before  the  five  years  of  this  Plan.  This  has  happened  to  three  projects:  Flexible  and 
Intelligent  Learning  Architectures  for  Systems  of  Systems  (FILA-SoS),  Agile  SE  Enablers  and  Quantification, 
and  Systems  Engineering  Expert  Knowledge  (SEEK).  Moreover,  one  project,  Body  of  Knowledge  and 
Curriculum  to  Advance  Systems  Engineering  (BKCASE),  was  in  the  original  Technical  Plan,  but  finished  its 
primary  research  phase  soon  after  the  original  Technical  Plan  was  approved.  Consequently,  BKCASE  never 
received  Core  funding,  even  though  it  continues  quite  successfully  to  this  day  in  operations  and 
maintenance. 

When  projects  end  earlier  than  planned,  freed  up  Core  funds  are  accumulated  in  an  investment  pool  that 
will  be  used  to  fund  new  programs  and  projects.  This  reduction  incentivizes  researchers  to  seek  additional 
non-Core  funding.  By  the  end  of  2018,  the  SERC  is  targeting  three  non-Core  dollars  for  each  Core-funded 
dollar. 

Additionally,  to  encourage  Pis  to  transition  their  research  results  into  university  courses,  some  Core  funds 
may  be  allocated  for  Pis  to  develop  educational  materials  based  on  their  research  results.  That  material 
will  be  shared  with  all  SERC  Collaborators  and  perhaps  more  broadly.  The  funding  level  and  timing  for  this 
allocation  is  yet  to  be  determined. 

Sections  5.1  through  5.4  describe  the  Core-funded  programs  and  projects  in  each  of  the  four  research 
areas  and  provide  a  short  summary  of  non-Core  funded  projects  in  those  areas  as  well.  Section  5.5 
describes  supporting  activities  that  enable  the  successful  execution  of  these  research  projects. 


5.1  Enterprises  and  Systems  of  Systems  (ESOS)  Research  Area 

Each  DoD/IC  Service  and  Agency,  and  the  larger  DoD  itself,  is  an  example  of  an  enterprise  with  all  the 
features  of  an  SoS.  Such  organizations  have  the  challenge  of  integrating  and  evolving  multiple  portfolios 
of  systems  with  often-conflicting  sets  of  objectives,  constraints,  stakeholders,  and  demands  for  resources. 
SoSs  generally  involve  integrating  multiple,  independently  managed  systems  to  achieve  a  unique 
capability,  therefore  involving  needs  for  collaboration  and  negotiation  as  well  as  control.  Thus,  when 
viewed  as  involving  both  the  technical  systems  and  their  organizational  management,  SoSs  are  enterprise 
challenges  as  well.  Indeed,  both  enterprises  as  systems  and  as  SoSs  increasingly  face  situations  in  which 
the  classical  systems  approach  of  deterministically  engineering  the  system  based  on  relatively  static 
requirements  and  specified  human  interactions  are  insufficient.  In  such  complex  systems,  human 
behavioral  and  social  phenomena  in  collaboration  are  critical  as  are  cascading  impacts  from 
interdependencies;  altogether,  emergent  outcomes  are  the  norm.  Research  is  necessary  to  determine  the 
foundational  SE  principles  for  such  systems.  These  principles  can  then  be  used  to  develop  associated  SE 
MPTs  applicable  to  such  complex  systems. 

ESOS  Area  Goal:  Through  the  use  of  advanced  MPTs,  transform  the  development  and 
delivery  of  end-to-end  defense  capability  to  DoD  service-providers  and  the  warfighters 
for  operation  in  complex  organizational  and  mission  environments,  so  those  capabilities 
have  with  fewer  unintended  negative  consequences  and  greater  resilience. 
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5.1.1  ESOS  Grand  Challenge  and  Current  Progress 

The  ESOS  Grand  Challenge  to  achieve  the  ESOS  Area  Goal  is  to: 

Create  the  foundational  SE  principles  and  develop  the  associated  MPTs  that  enable  the 
DoD  and  its  partners  to  model  (architect,  design,  analyze),  acquire,  evolve  (operate, 
maintain,  monitor)  and  verify  complex  enterprises  and  systems  of  systems  to  generate 
affordable  and  overwhelming  competitive  advantage  over  its  current  and  future 
adversaries. 

Researchers  in  the  ESOS  area  have  made  substantial  progress  towards  meeting  the  ESOS  Research  Grand 
Challenge,  particularly  in  the  ability  to  model  and  analyze  complex  interdependencies  and  the  ability  to 
apply  models  with  case  studies  to  guide  operations.  Examples  include  the  SoS  Analytic  Workbench,  the 
completed  FILA-SoS  body  of  work,  valued-based  Kanban  scheduling  for  SoS  capability 
development/enhancement,  and  the  use  of  enterprise  systems  modeling  (a  broader  notion  than  simply 
multi-level  models)  demonstrated  in  the  context  of  counterfeit  parts.  Some  adjacent  work  in  linking  SoS 
cost  models  to  architecture  evolution  via  the  Systems  Modeling  Language  (SysML)  has  also  advanced,  and 
should  be  more  tightly  integrated  with  other  SoS  activities  for  greater  impact.  Early  work  in  enterprise 
system  models  and  SysML  activity  point  to  the  need  for  even  greater  effort  in  visualization  and  direct  tools 
for  decision-support  during  operations  and  evolution. 

Gaps  remain  to  address  the  ESOS  Grand  Challenge,  especially  in  the  ability  of  individual  systems  to 
understand  implications  from  the  SoS  architecture  and  its  behaviors.  A  greater  "situational  awareness" 
for  key  systems  would  increase  their  ability  to  thrive  in  the  highly  dynamic  and  emergent  nature  of  SoS 
and  Enterprises.  Collaborative  decision-making  tools  are  promising  in  this  regard.  Also,  concepts  such  as 
quantifying  technical  debt  in  existing  systems  could  provide  a  means  for  this  situational  awareness  and 
understanding  systems  that  may  support  a  new  or  evolved  capability. 

In  addition,  although  the  SoS  Analytic  Workbench  and  the  counterfeit  parts  enterprise  model  have  been 
successfully  demonstrated  in  the  SERC  Innovation  and  Demonstration  Laboratory  (SIDL)  (see  Section  5.5 
for  more  on  this  Laboratory),  gaps  still  remain  in  achieving  more  ubiquitous  and  flexible  availability  to  DoD 
communities.  The  ESOS  Goal  requires  that  the  DoD  actually  use  models  and  tools  to  make  the  necessary 
decisions  that  lead  to  superior  outcomes.  This  supports  the  vision  for  a  form  of  "SoS  Engineering  tool 
repository"  that  will  be  hosted  in  the  SIDL.  Such  a  repository  would  identify  the  right  tools  available  for  a 
particular  problem,  where  they  can  be  found,  and  how  they  can  be  used  in  proper  context.  While  this 
would  not  prohibit  inventors  from  advancing  and  distributing  the  tools  in  other  ways,  a  repository  with 
administration  by  SERC  could  fill  a  gap  until  such  tools  come  to  market. 


5.1.2  Strategies  to  Address  the  ESOS  Grand  Challenge 

Successfully  executing  the  following  strategies  will  make  significant  progress  towards  addressing  the  ESOS 
Grand  Challenge: 

1.  Model:  Develop  MPTs  that  allow  quick  and  insightful  modeling  of  enterprises/SoSs  so  that  the 
effects  of  changes  in  policies,  practices,  components,  interfaces,  and  technologies  can  be 
anticipated  and  understood  in  advance  of  their  implementation 
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2.  Acquire:  Develop  MPTs  that  allow  insight  into  enterprise/SoS  capability  acquisition  approaches 
in  the  face  of  significant  uncertainty  and  change  to  minimize  unintended  consequences  and 
unforeseen  risks 

3.  Evolve:  Develop  MPTs  that  facilitate  evolving  and  growing  an  enterprise/SoS,  including  insight 
into  different  architectural  integration  and  collaboration  approaches  that  facilitate  evolution  in 
the  face  of  uncertainty  and  change  in  how  an  enterprise/SoS  is  employed,  the  technologies 
available  to  realize  it,  and  the  environment  in  which  it  exists 

4.  Verify:  Develop  MPTs  that  allow  the  properties  of  an  enterprise/SoS  to  be  anticipated,  monitored 
and  confirmed  during  development  and  evolution,  including  an  enterprise/SoS  which  includes 
legacy  systems  that  are  in  operation  while  development  and  evolution  are  underway 

Recall  that,  for  the  SERC,  development  of  an  MPT  includes  validation  and  transition. 

Directly  implementing  the  strategies  are  two  research  programs,  described  below:  Enterprise  Modeling 
and  Systems  of  Systems  Modeling  and  Analysis. 


5.1.3  Enterprise  Modeling  Program 

This  program  has  been  focused  on  developing  a  rigorous  systems  science  and  engineering  foundation  for 
ESOS  and  on  building  a  community  of  researchers  who  collectively  will  advance  ESOS  research.  In 
addition,  ESOS  researchers  have  been  developing  domain-specific  multi-level  or  multi-scale  models  in 
areas  such  as  counterfeit  parts,  healthcare  delivery,  and  urban  resilience  with  support  from  a  variety  of 
sponsors. 

This  foundation  and  experiences  have  provided  the  basis  to  take  on  the  broader  goal  of  providing 
enterprise  level  process  mapping,  monitoring  and  control  for  the  ESOS  Grand  Challenge.  As  shown  in 
Figure  5.1-1,  data,  forecasts  and  reports  (including  text)  flow  from  the  context  of  an  enterprise's  ongoing 
transformation;  data  and  text  mining  are  used  to  make  sense  of  this  flow  of  information;  the  insights 
gained  enable  computational  modeling;  the  results  of  which  are  used  to  drive  interactive  visualizations 
that  enable  process  mapping,  monitoring  and  control. 

Pursuit  of  the  ESOS  Grand  Challenge  represents  a  transition  from  focusing  solely  on  the  design  of 
enterprises  and  systems  of  systems  to  the  design  and  operation  of  enterprises  and  systems  of  systems.  In 
other  words,  the  computational  models  and  interactive  visualizations  would  be  run  in  parallel  with  actual 
operations  and  provide  a  means  for  monitoring  and  the  control  of  operations.  In  short,  the  goal  is  to 
provide  model-based  enterprise  diagnostics  and  decision  support.  Key  to  this  approach  is  the 
understanding  that  mathematical  and  computational  models  cannot  capture  everything  that  is  relevant 
to  the  performance  of  an  enterprise.  Consequently,  designing  mechanisms  that  enable  adaptation  and 
hedging  is  critical.  This  would  enable  the  operators  of  complex  systems  to  detect,  diagnose,  and 
compensate  for  deviations  of  operations  from  expectations.  The  complex  enterprise  systems  of  interest 
could  range  from  military  operations  and  acquisition  programs,  to  urban  infrastructures  and  healthcare 
delivery  organizations. 
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Figure  5.1-1.  Enterprise  Process  Mapping,  Monitoring  and  Control  Communication  Flows 


This  research  program  implements  all  four  ESOS  strategies  and  so  far  includes  one  project:  Enterprise 
Systems  Analysis  (formerly  known  as  Multi-Level  Modeling  ofSocio-Technical  Systems).  Table  5.1-1  offers 
a  description  of  this  project  and  which  strategies  it  supports. 


Table  5.1-1.  Projects  in  the  Enterprise  Modeling  Program 


Project 

Started 

Purpose 

Primary  ESOS 
Supported  Strategies 

Enterprise 
Systems  Analysis 

2012 

Develop  MPTs  to  model,  understand,  and  evolve 
enterprise  systems 

1,  2,  3,  4 

5. 1.3.1  Enterprise  Systems  Analysis  Project 

Analyzing  enterprise  systems  often  requires  modeling  them  on  several  different  levels  of  abstraction.  For 
instance,  DoD  spends  billions  of  dollars  annually  on  its  modeling  and  simulation  programs10,11’12.  These 
models  and  simulations  include  million  lines  of  code  and  model  everything  from  weapon  platforms  to 
operational  military  organizations.  Each  level,  from  the  performance  of  individual  weapon  systems  to  the 
success  of  a  campaign,  is  important,  and  they  all  interact.  Presumably,  the  reason  one  develops  better 
weapons  is  to  improve  success  on  the  battlefield  and  increase  national  security.  Consequently,  the 
seemingly  obvious  solution  to  model  the  DoD  enterprise  would  be  to  link  low-level  models 
computationally  to  the  high-level  models.  Unfortunately,  such  models  rarely  interact  with  each  other 
without  substantial  human  intervention. 

The  Enterprise  Systems  Analysis  Project  originally  began  as  the  Multi-Level  Modeling  of  Socio-Technical 
Systems  Project  with  the  aim  of  enhancing  capabilities  to  allow  models  of  different  levels  of  fidelity  and 


10  See  http://sim-summit.org/BusinessPractice/pdfs/RQI%20Final%20Report.pdf. 

11  See  http://msco.mil/documents/MSJournal2012Fall.pdf. 

12  K.  Baldwin  and  J.  Citizen,  "Modeling  and  Simulation  Technological  and  Industrial  Base",  a  briefing  on  the  FY15 
NDAA  Sec  223,  July  30,  2015. 
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abstraction  to  interoperate  in  a  dynamic  fashion.  However,  as  the  work  on  this  effort  proceeded,  the 
understanding  of  the  problem  has  evolved.  While  multi-level  models  of  enterprises  are  important,  models 
at  different  levels  of  abstraction  cannot  always  be  connected  computationally.  These  connections  depend 
critically  on  the  enterprise  context  and  question  being  asked.  As  a  result,  this  introduces  a  number  of  risks 
when  one  attempts  to  make  decisions  based  on  such  models.  Consequently,  enterprises  systems  analysis 
has  evolved  to  emphasize  using  multi-level  models  to  understand  the  risks  inherent  to  analyzing, 
designing  and,  operating  enterprise  systems  and  then  enabling  decision  makers  to  better  manage  those 
risks.  This  will  be  accomplished  by  developing  methods  to  enable  analysts  and  decisions  makers  to: 

•  Understand  when  and  how  enterprise  systems  can  be  modeled  computationally 

•  Understand  the  risks  imposed  by  the  aspects  of  the  enterprise  that  cannot  be  modeled 
computationally 

•  Explore  the  associated  tradeoffs  through  the  use  of  interactive  visualization 

•  Design  mechanisms  to  adapt  to  and  hedge  the  risks. 

The  overall  approach  is  to  explore  the  problem  space  through  a  series  of  case  studies  that  result  in  the 
development  of  enterprise  models  that  analyze  problems  of  interest  to  DoD.  The  first  case  study  involves 
the  development  of  a  multi-level  model  of  counterfeit  part  intrusion  into  defense  supply  chains.  By  using 
multiple  case  studies,  methods  can  be  developed  and  evaluated  in  successive  iterations  over  a  broad 
range  of  phenomena. 

To  date,  the  counterfeit  parts  case  study  has  involved  multiple  roundtable  sessions  that  brought  together 
subject  matter  experts  from  both  government  and  industry.  The  resulting  model  was  validated  through  a 
series  of  reviews  and  is  continuing  to  be  refined  and  enhanced.  The  intent  is  to  transition  the  simulation 
to  a  government  customer  by  the  end  of  2016. 

Work  under  this  project  has  led  to  a  new  textbook,  "Modeling  and  Visualization  of  Complex  Systems  and 
Enterprises,"  which  was  published  by  John  Wiley  in  July  2015.  It  includes  a  ten-step  methodology, 
recommended  methods  and  tools,  and  a  review  of  a  range  of  modeling  paradigms.  The  book  includes 
many  examples  and  case  studies,  ranging  from  counterfeit  parts  in  DoD  supply  chains  to  archetypal 
problems  in  healthcare  delivery,  cancer  biology,  traffic  congestion,  urban  resilience  and  financial  bubbles. 
The  book  was  used  in  a  course  of  the  same  name  in  the  spring  2015  term  at  Steven's  School  of  Systems 
and  Enterprises  with  11  PhD  students  enrolled,  all  of  whom  created  demonstrations  of  interactive 
enterprise  models.  These  outcomes  are  evidence  of  transitions  of  this  project  to  systems  engineering 
practice. 

Table  5.1-2  shows  the  focus  and  deliverables  in  Enterprise  Systems  Analysis  through  2018. 


Table  5.1-2.  Enterprise  Systems  Analysis  Project  Timeline* 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Architectural  Assessment  of  existing 

Technical  reports,  summary  of  interviews  and 

commercial  practices 

observations 

2014  -  2015 

Develop  modeling  methods  and  strategies 

Technical  Reports,  Counterfeit  Parts  Simulation 

2016  -  2018 

Validate  modeling  methods  and  strategies 

Technical  Reports,  2nd  Case  Study  Simulation 
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The  Enterprise  Systems  Analysis  Project  transition  action  plan  and  characterization  are  shown  in  Tables 
5.1-3  and  5.1-4  below. 


Table  5.1-3.  Enterprise  Systems  Analysis  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Held  a  series  of  roundtable  sessions  with  key  subject  matter  experts  and 
stakeholders  concerned  with  the  counterfeit  parts  problem  to  guide 
simulation  development 

•  Engage  Community 

•  Pilot  Continuously 

2 

Conduct  a  workshop  on  the  counterfeit  parts  simulation  with  key 
stakeholders  from  industry  and  government  to  demonstrate  the  value  of  the 
enterprise  view  for  policymaking  and  identify  an  adopter.  Refine  the 
simulation  based  on  feedback  and  structure  for  transition. 

•  Engage  Community 

•  Productize 

3 

Employ  a  series  of  incremental  DoD  application-oriented  case  studies  to 
explore  long-term  theoretical  implications  while  spinning  off  useful 
increments  to  practice. 

•  Plan  Early 

•  Balance  Long  and 

Short  term 

•  Pilot  Continuously 

4 

Present  theoretical  findings  at  conferences  and  publish  in  peer-reviewed 
journals  to  solicit  feedback,  validate  findings,  and  disseminate  work. 

•  Engage  Community 

Table  5.1-4.  Enterprise  Systems  Analysis  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Stevens  Institute  of  Technology  now  teaches  a  graduate  level  class  that  incorporates 
findings  from  this  research  effort 

Progress 

(approval, 

adoption) 

•  The  Project  has  resulted  in  4  technical  reports,  3  conference  papers,  1  published 

textbook,  and  1  submitted  journal  article 

•  The  Project  will  hold  both  a  SERC  peer  review  of  the  counterfeit  parts  simulation  and 

workshop  with  stakeholders  from  industry  and  government 

5.1.4  Systems  of  Systems  Modeling  and  Analysis  Program 

Developing  new  SoS  capabilities  while  evolving  SoSs  over  time  to  improve  performance  and  stay  current 
with  new  technologies  remains  highly  challenging.  The  complex  interdependencies  among  systems  often 
exhibit  managerial  and  operational  independence,  yet  must  work  cohesively  to  achieve  an  overarching 
set  of  capabilities.  Tradeoffs  between  capability  and  risk  are  essential  decisions  that  must  be  addressed 
for  SoS  capability  planning.  Existing  tools  for  such  tradeoffs  are  of  limited  value  when  size  and/or 
interdependency  complexity  is  high. 

This  research  program  addresses  the  need  to  create  and  mature  decision-support  tools  specifically  for 
evolving  SoS  architectures  and  capabilities.  The  research  to  date  has  explored  analytical  methods  to 
quantify  the  impact  of  system  interdependencies  in  the  context  of  SoS  capability  development  as  well  as 
broader  agent  models  that  address  the  often-fuzzy  influence  of  stakeholder  perspectives  in  the  technical 
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development  activities.  Additional  research  has  focused  on  identifying  innovative  approaches  to  support 
SE  in  architecting,  engineering,  and  evolving  complex  SoS.  Continuing  research  in  this  area  will  focus  on 
SoS  and  constituent  system  situational  awareness,  strategic  approaches  for  simplifying  SoS  architectures 
and  their  ability  to  restructure  quickly  to  respond  to  new  needs  and  missions,  as  well  as  the 
implementation  of  an  SoS  Toolbox  repository  to  make  maturing  SoSE  tools  generally  available  to  SoS  and 
constituent  system  development  teams. 

This  research  program  implements  all  four  ESOS  strategies  and  will  address  ESOS  Strategy  4,  Verify,  with 
more  vigor  as  pilot  studies  are  completed.  It  includes  one  project  still  underway  (Systems  of  Systems 
Analytic  Workbench )  and  one  just  successfully  completed  ( Flexible  Intelligent  Learning  Architectures  for 
Systems  of  Systems  (FILA-SoS)).  FILA-SoS  produced  software  that  is  currently  being  transitioned  via 
experimentation  in  the  SERC  Innovation  &  Demonstration  Lab  and  also  through  the  annual  Complex 
Adaptive  Systems  Conference13. 

Table  5.1-5  offers  a  description  of  both  projects  and  which  strategies  they  primarily  support. 

Table  5.1-5.  Projects  in  the  System  of  Systems  Modeling  and  Analysis  Program 


Project 

Started 

Purpose 

Primary 

ESOS 

Supported 

Strategies 

Systems  of  Systems 
Analytic  Workbench 

2011 

Develop  MPTs  and  an  Analytic  Workbench  construct  to 
house  them  for  the  purpose  of  SoS  architecture  analysis, 
redesign  and  evolution  management 

1,  2,  3,  4 

Flexible  Intelligent 
Learning  Architectures 
for  Systems  of  Systems 
(FILA-SoS) 

2011 

(completed 
successfully 
Jan.  2015) 

Developed  proof  of  concept  decision  making  tool  based 

on  the  Wave  Process  Model  for  architecture  selection 
and  evolution;  the  tool  is  to  be  used  by  an  Acknowledged 
System  of  Systems  Manager 

1,  2,  3,  4 

5. 1.4.1  Systems  of  Systems  Analytic  Workbench  Project 

The  objective  of  this  project  is  to  develop  an  SoS  Analytic  Workbench  (AWB).  The  AWB  is  an  organized  set 
of  computational  tools  that  can  aid  practitioners  in  making  decisions  on  evolving  SoS  architectures  and 
understanding  complex  interdependencies.  Typical  questions  asked  by  SoS  practitioners  have  been 
collected  by  the  project  team  and  mapped  to  methods/formulations  appropriate  to  produce  the  desired 
analytical  outputs.  A  key  emphasis  in  the  workbench  approach  is  to  relegate  the  difficult  complexities  in 
dealing  with  highly  interconnected  systems  within  an  SoS  to  the  methods,  while  empowering  the  decision 
maker  with  the  products  expressed  in  understandable  tradespace  visualizations.  The  methods 
demonstrated  so  far  include:  Robust  Mean  Variance  Optimization,  Systems  Operational  Dependency 
Analysis  (extended  and  enhanced  from  Functional  Dependency  Network  Analysis  developed  at  MITRE)), 
Approximate  Dynamic  Programming,  and  System  Importance  Measures  (a  means  for  assessing  and 
selecting  systems  based  on  resilience  properties  brought  to  the  SoS).  In  addition,  progress  has  been  made 
in  developing  a  common  input  data  model  from  which  each  method  can  operate  with  as  little  tailoring  as 
possible.  To  evaluate  the  progress,  a  team  of  analysts  from  MITRE  undertook  a  series  of  pilot 


13  See  http://complexsystems.mst.edu/. 
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demonstrations  of  the  AWB;  their  report  documented  useful  directions  for  future  enhancements  as  well 
as  assessment  of  the  value  brought  by  such  a  capability. 

Table  5.1-6  shows  the  focus  and  deliverables  in  the  Systems  of  Systems  Analytic  Workbench  Project 
expected  through  2018. 


Table  5.1-6.  SoS  Analytic  Workbench  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core** 

Evaluate  architectures  and  develop  proof  of 
concept  simulation 

Technical  reports  and  agent  based  model  for 
selected  application  domain 

2014  -  2015 

Develop  prototype  tools 

Technical  reports,  demonstration  of  prototype 

2016  -  2018 

SoS  AWB  Project  (end  Fall  2016):  Complete 
pilot  studies  and  integration  in  the  SERC 
Innovation  and  Demonstration  Lab 

Technical  reports  and  articles  that  document 
efficacy  and  use  of  the  AWB  MPTs;  complete, 
demonstration  of  prototype  application 

**Note:  Funding  shown  from  Pre-Core  through  2014-15  included  both  SoS  Analytic  Workbench  and  FILA-SoS  projects 


In  addition  to  the  SoS  Analytic  Workbench  itself,  several  other  tools  and  techniques  have  been  developed 
or  identified  to  support  SoS  architecture  assessment,  architecture  evolution,  and  capability  engineering 
and  development  -  some  as  a  result  of  other  SERC  research  areas.  To  date,  information  about  these  tools 
and  techniques  primarily  resides  in  SERC  and  other  technical  reports.  In  2016,  efforts  will  begin  to  catalog 
these  tools  and  techniques  in  a  repository  and  include  copies  of  any  tools  that  have  been  developed  as 
part  of  SERC  research  or  other  non-Core  SoS  funding  that  can  be  made  generally  available  to  the  SERC  and 
engineering  communities.  Once  these  tools  have  been  identified  and  cataloged,  further  analysis  can  be 
conducted  with  respect  to  coverage  of  System  of  Systems  Engineering  (SoSE)  activities/tasks,  gaps  with 
respect  to  difficult  or  complex  SoSE  activities,  and  interoperability  of  tools,  along  with  guidance  on  how 
tools  can  be  used  to  either  inform  or  address  SoS  problems,  issues,  and  concerns. 

The  SoS  Analytic  Workbench  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.1-7 
and  5.1-8  below. 


Table  5.1-7.  SoS  Analytic  Workbench  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

The  SoS  Analytic  Workbench  has  been  studied,  exercised,  and  evaluated  in  a 
pre-pilot  phase  by  MITRE  Corporation.  Pre-pilot  activities  have  been 
conducted  with  Army  Research  Lab  analysts  for  application  to  live,  virtual, 
constructive  test  design. 

•  Pilot  Continuously 

•  Engage  Community 

•  Productize 

2 

Demonstrations  and  discussions  ongoing  with  Johns  Hopkins  Applied  Physics 

Lab  (JHUAPL)  and  the  US  Air  Force.  To  support  these  activities,  graphic  user 
interfaces  and  input  test  cases  have  been  developed  and  refined. 

•  Pilot  Continuously 

•  Engage  Community 

•  Productize 

3 

A  CRADA  between  Purdue  University  and  Navy  NSWC  Dahlgren  has  been 
signed  and  is  being  applied  in  which  Navy  experts  will  conduct  exploratory 
pilot  applications  of  the  SoS  AWB  as  it  relates  to  enhancement  of  Navy 
Interoperability  &  Integration  studies. 

•  Plan  Early 

•  Pilot  Continuously 
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Table  5.1-8.  SoS  Analytic  Workbench  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Purdue  University  teaches  a  graduate  level  class  (AAE560  Systems  of  Systems  Modeling 
and  Analysis)  that  has  been  significantly  enhanced  by  the  findings  from  this  research 
effort 

Progress 

(approval, 

adoption) 

•  Several  journal  and  conference  papers  have  been  generated,  including  a  recent 
publication  in  CSER  2015  and  a  paper  in  the  INCOSE  Systems  Engineering  Journal 

5. 1.4.2  Flexible  Intelligent  Learning  Architectures  for  Systems  of  Systems  (FILA-SoS)  Project 

The  FILA-SoS  Project  developed  an  integrated  model  based  on  the  Wave  Process  Model  which  serves  as  a 
decision  making  aid  for  an  SoS  manager.  The  FILA-SoS  model  does  so  by  using  a  straightforward  system 
definitions  methodology  and  an  efficient  analysis  framework  that  support  rapid  exploration  and 
understanding  of  key  trade-offs  and  requirements  for  a  wide  range  of  SoS  stakeholders  and  decision 
makers.  The  FILA-SoS  model  addresses  four  of  the  most  challenging  aspects  of  SoS  architecting: 

1.  Dealing  with  the  uncertainty  and  variability  of  the  capabilities  and  availability  of  potential 
component  systems 

2.  Providing  for  the  evolution  of  the  SoS  needs,  resources  and  environment  over  time 

3.  Accounting  for  the  differing  approaches  and  motivations  of  the  autonomous  component  system 
managers 

4.  Optimizing  SoS  characteristics  in  an  uncertain  and  dynamic  environment  with  fixed  budget  and 
resources 

Three  notional  SoSs  have  been  modeled:  Conceptual  Problem  for  aircraft  carrier  performance 
assessment,  for  intelligence  surveillance  and  reconnaissance,  and  for  search  and  rescue. 

As  shown  in  Figure  5.1-2,  the  FILA-SoS  framework  offers  several  unique  capabilities  that  aid  the  SoS 
manager  for  a  variety  of  complex  systems  such  as  logistics  and  cyber-physical  systems.  These  capabilities 
include: 

•  Being  an  integrated  approach  to  model  and  simulate  SoS  systems  that  incorporate  evolution  for 
multiple  waves, 

•  Having  a  modular  structure  that  enables  models  to  be  run  independently  or  in  conjunction  with 
each  other, 

•  Offering  models  for  both  architecture  generation  and  SoS  behavior  as  well  as  models  to  negotiate 
system  behavior  between  SoS  and  individual  systems, 

•  Helping  understand  emergent  behavior  of  systems  in  the  acquisition  environment  and  their 
impact  on  SoS  architecture  quality, 

•  Serving  as  a  means  to  study  the  dynamic  behavior  of  different  types  of  systems  -  non-cooperative, 
semi-cooperative,  and  cooperative, 
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•  Enabling  identification  of  intra-  and  inter-dependencies  among  SoS  elements  and  the  acquisition 
environment, 

•  Providing  a  "what-if"  analysis  relying  on  variables  such  as  SoS  funding  and  capability  priority; 
those  variables  can  be  changed  as  the  acquisition  progresses  through  wave  cycles, 

•  Simulating  any  architecture  through  colored  petri  nets, 

•  Simulating  rules  of  engagement  and  behavior  settings  where  there  is  a  mix  of  whether  systems 
are  non-cooperative,  semi-cooperative,  or  cooperative,  and 

•  Acting  as  a  test-bed  for  decision  makers  to  evaluate  operational  guidelines  and  principles  for 
managing  various  acquisition  environment  scenarios. 

Future  capabilities  include  extending  the  modeling  capability  to  accommodate  multiple  interface 
alternatives  among  systems  and  incorporating  risk  models  into  environmental  scenarios. 


Initialize  SoS 


s°< , 

Architectui 


Meta 

architecture 

generated 


Cond1  't  SoS  Analysis 


Object  Proi 
Methodoli 


Develop/Evolve  SoS 


Colored  F 
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System 

Behavior  Types 


Executable  Models 


Figure  5.1-2.  FILA-SoS  Capabilities 

Table  5.1-9  shows  the  focus  and  deliverables  in  the  FILA-SoS  Project  through  its  completion. 
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Table  5.1-9.  FILA-SoS  Project  Timeline* 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Develop  the  basic  structure  of  the  FILA-SoS 
model  based  on  the  Wave  Process  Model  and 

develop  mathematical  models  for  each 
independent  component 

Technical  reports,  journal  and  conference 
publications.  Prototype  software  to  demonstrate 
the  workings  of  the  FILA-SoS  model  on  two 
notional  SoSs:  (1)  Surveillance,  Intelligence  and 
Reconnaissance,  and  (2)  Search  and  Rescue 

2014 

Development  of  algorithms  that  may  be 
integrated  into  FILA-SoS  to  answer  research 
questions  on  subsystem  behavior  and  on  how 
to  incentivize  systems  to  participate  in  an  SoS 

17  volumes  of  technical  reports,  journal  and 
conference  publications,  FILA-SoS  Version  1.0 
Scalability  Validation  with  MITRE  Data,  and 
demonstration  of  the  integrated  model  though  a 
conceptual  problem  of  assessing  the  performance 
of  an  aircraft  carrier 

The  FILA-SoS  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.1-10  and  5.1-11 
below. 


Table  5.1-10.  FILA-SoS  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

The  FILA-SoS  model  was  presented  to  "SoS  Concept  Development  and 

Assessment,  SoSITE"  at  Boeing  Company  and  the  approached  was  transferred  to 
Boeing  Engineering  by  a  Missouri  S&T  Systems  Engineering  PhD  student. 

•  Productize 

•  Engage  Community 

2 

The  FILA-SoS  model  is  being  integrated  to  other  research  projects  in  the  ESOS 
research  area  in  the  SERC  Innovation  and  Demonstration  Laboratory  using  the 
conceptual  problem  for  assessing  the  performance  of  the  aircraft  carrier.  It  is 
also  possible  to  use  this  new  facility  to  demonstrate  the  FILA-SoS  model  to 
interested  parties  for  its  implementation  to  a  specific  SoS  or  cyber-physical 
system. 

•  Engage  Community 

•  Productize 

3 

Fall  2015  Missouri  University  of  Science  and  Technology  SySEng  6239  was 
completely  restructured  to  accommodate  algorithms  and  integrated  model 
structure  developed  in  FILA-SoS  research.  The  first  version  of  FILA-SoS 
prototype  software  was  the  basis  of  small  project  tasks  to  be  conducted  during 

the  course.  The  intent  is  to  make  these  course  materials  available  at  the  Naval 
Postgraduate  School  and  the  Air  Force  Institute  of  Technology  to  be  used  in 
conjunction  with  the  SERC  Innovation  and  Demonstration  Lab  during  Spring  or 

Fall  2016  in  similar  courses  that  they  teach. 

•  Plan  Early 

•  Engage  Community 

•  Productize 

4 

Successfully  completed  FILA-SoS  project  has  spurred  the  development  and 
growth  of  the  Complex  Adaptive  Systems  Conference,  along  with  the  CSER,  a 
significant  community  building  and  sharing  forum  to  transition  ideas  and  tools 

•  Engage  Community 

•  Balance  Long  & 

Short  Term 
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# 

Transition  Action 

Principles  Implemented 

5 

The  second  version  of  FILA-SoS  prototype  software  (beyond  the  scope  of  this 
project)  will  incorporate  models  developed  to  answer  stated  research  questions. 
This  requires  professional  software  development,  as  research  software 
prototype  is  not  sufficient  for  this  purpose.  It  may  be  possible  to  work  with 
software  companies  or  John  Flopkins  Applied  Physics  Lab  for  possible  transfer  of 
the  research  to  be  developed  further  for  industrial  use. 

•  Productize 

Table  5.1-11.  FILA-SoS  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Missouri  University  of  Science  and  Technology  teaches  graduate  classes  (SySEng  6104 
Systems  Architecting  and  SySEng  6239  Smart  Engineering  System  Designs)  that  have 
been  significantly  enhanced  by  the  findings  from  this  research  effort. 

Progress 

(approval, 

adoption) 

•  Nineteen  journal  and  conference  papers  have  been  generated  based  on  this  research 
project  and  the  annual  Complex  Adaptive  Systems  Conference  was  and  will  continue  to 
be  a  good  outlet  for  diffusion  of  this  research. 

5.1.5  ESOS  Area  Non-Core  Funded  Projects 

During  the  time  of  the  Technical  Plan,  the  SERC  has  been  awarded  one  ESOS  non-Core  funded  project. 
The  project,  which  is  briefly  described  in  Table  5.1-12,  is  still  active  as  of  the  time  of  the  publication  of  this 
Plan. 


Table  5.1-12.  ESOS  Area  Non-Core  Funded  Project 


Project 

Sponsor 

Description 

Army  Lethality 
Study 

OSD 

Perform  a  systems-oriented  study  to  assess  the  current  state  of  the  Army's  lethality 
capability  and  provide  actionable  recommendations  to  make  the  changes  needed 
to  transform  that  capability  at  an  enterprise  level. 

5.2  Trusted  Systems  (TS) 

The  organization  of  its  assets  into  net-centric  systems  of  systems  (NCSOS)  has  enabled  DoD  to  much  more 
rapidly  and  effectively  see-first,  understand-first,  act-first,  and  finish  decisively  in  its  operations.  However, 
this  implies  that  each  of  its  assets  needs  to  achieve  higher  levels  of  trust  as  part  of  the  NCSOS,  as  compared 
to  its  previous  role  as  a  standalone  platform,  all  the  while  retaining  or  improving  its  previous  speed  and 
effectiveness.  Achieving  those  levels  of  trust  is  extremely  challenging. 

The  SERC  Trusted  Systems  (TS)  research  area  addresses  this  challenge,  in  part,  by  balancing  traditional 
reactive  cyber-security  defenses  with  pro-active  mechanisms,  making  attacks  more  difficult  and  more 
expensive.  It  combines  this  approach  with  increased  and  continuous  application  of  advanced  assurance 
capabilities  that  concurrently  address  not  only  security  but  also  safety,  reliability,  availability, 
maintainability,  usability,  interoperability,  and  resilience. 
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TS  Area  Goal:  Transform  system  assurance  from  a  late,  reactive  activity  into  an  early  and 
continuous,  pro-active  orchestration  of  advanced  assurance  MPTs,  in  ways  that  balance 
the  simultaneous  achievement  of  cyber-security  trust  and  assurance  with  complementary 
MPTs  for  assuring  safe,  reliable,  available,  usable,  interoperable,  and  resilient  mission 
cost-effectiveness. 


5.2.1  TS  Grand  Challenge  and  Current  Progress 

The  TS  Grand  Challenge  to  achieve  the  TS  Area  Goal  is  to: 

Achieve  much  higher  levels  of  system  trust  by  applying  the  systems  approach  to  achieving 
system  assurance  and  trust  for  the  increasingly  complex,  dynamic,  cyber-physical-human 
net-centric  systems  and  systems  of  systems  of  the  future. 

With  respect  to  security,  this  activity's  mainline  concept  of  adding  a  layer  of  security  through  securely 
monitoring  systems  for  system  illogical  behaviors  that  can  be  assessed  as  most  likely  caused  by  a 
cyberattack  has  received  significant  recognition,  both  within  and  outside  of  the  DoD.  The  approach  uses 
a  highly  secured  Sentinel  as  both  a  valuable  addition  to  security  and  as  an  economically  advantageous 
system  architecture,  compared  to  directly  securing  the  monitored  system  to  a  similar  level  of  security  in 
the  Sentinel.  In  particular,  application  to  physical  systems  has  been  seen  as  an  important  opportunity  for 
application  of  this  technological  approach,  as  the  development  and  securing  of  a  Sentinel  can  include 
elements  such  as  independent  sensors  and  bounded  operator  control  rules  as  the  basis  for  effective  and 
economical  design  approaches  for  detecting  attacks.  This  has  resulted  in  the  start  of  prototyping  projects 
for  a  DoD  radar,  a  3D  printer  (NIST),  police  cars  (Virginia  State  Police),  and  an  Army/Air  Force  image 
exploitation  system  using  a  private  cloud-based  Sentinel. 

In  the  process  of  conducting  this  research,  two  important  gaps  in  the  needs  for  security  have  been 
recognized.  First,  in  military  operations,  individual  systems  are  clustered  into  SoSs  that  conduct  missions. 
The  missions  are  the  capabilities  that  need  added  security,  while  the  individual  systems  are  the  specific 
locations  for  providing  the  elements  of  security.  We  have  seen  that  in  an  SoS:  (1)  attacks  can  be  developed 
to  exploit  the  seams  between  individual  systems;  (2)  attacks  occurring  in  a  particular  system  can  result  in 
symptoms  that  appear  in  another;  and  (3)  defense  alternatives  of  certain  mission  functions  that  can  be 
addressed  in  more  than  one  system,  can  sometimes  be  much  more  easily  and  economically  accomplished 
in  one  system  as  compared  to  another  system.  This  leads  to  the  recognition  of  the  need  for  better 
understanding,  analysis  and  design  for  mission-level  security.  In  2014-2015  the  SERC  developed  an  SoS 
test  bed  including  multiple  ground-target  sensor  types  (video,  infrared,  acoustic),  a  radar  for  airborne 
targets,  an  unmanned  aerial  vehicle,  an  image  exploitation  system  and  a  ground  defense  command  and 
control  system.  The  Secure  Mission  Laboratory  enables  the  start  of  a  research  effort  that  expands  the 
Sentinel  concept  to  the  mission  level,  including  multiple  Sentinels  providing  the  basis  for  managing 
cyberattacks  from  a  mission  perspective.  A  partnership,  initiated  through  the  Naval  Cyber  Command,  with 
JHUAPL  has  started  working  on  new  concepts  for  mission  level  security,  with  the  objective  of  engaging 
operational  military  leaders  into  the  process  of  mission  security  requirements,  and  engaging  the 
technology  community  into  new  approaches  that  more  directly  relate  solutions  to  mission  performance. 

Second,  the  emergence  of  new  DoD  initiatives  related  to  highly  automated/autonomous  systems  drives  a 
need  for  more  advanced  risk  containment  technologies.  These  technologies  include  both  cybersecurity 
and  other  technologies  that  offer  resilience  that  assures  continuous  system  operation.  The  Sentinel 
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approach  has  clear  overlaps  with  other  risk  containment  approaches  (e.g.,  fault  tolerant  design,  software 
assurance  solutions),  providing  an  opportunity  to  integrate  these  opportunities  from  (1)  a  design  concept 
viewpoint,  (2)  an  implementation  viewpoint,  and  (3)  in  certain  cases,  from  an  integrated  implementation 
viewpoint.  In  addition  to  technology  design  issues,  other  issues  related  to  the  role  of  people  in  these 
systems  contain  significant  overlaps  as  well.  This  year,  the  cybersecurity  project  has  included  a  human 
factors  element,  addressing  human  confidence  in  responsive  decision-making  in  the  event  of  a 
cyberattack.  The  experimental  questions  being  addressed  in  the  human  factors  project  have  been  seen  to 
overlap  with  questions  faced  by  the  Air  Force  autonomous  systems  community  at  the  Air  Force  Research 
Laboratory.  The  Air  Force  Institute  of  Technology  is  engaged  with  the  University  of  Virginia  in  addressing 
the  cybersecurity  project  and  is  actively  evaluating  the  synergistic  opportunities  for  future  research.  The 
SERC  anticipates  identifying  important  overlapping  interests  and  defining  new  project  objectives  for  2016 
that  address  the  synergistic  opportunities. 

In  the  area  of  assurance,  this  goal  can  only  be  met  through  a  comprehensive  and  aggressive  SE  approach 
that  encompasses  three  critical  dimensions  of  consideration:  (1)  the  structure  of  systems,  including 
architecture  and  accounting  for  various  kinds  of  dynamism  for  the  purpose  of  resiliency  and  autonomy, 
(2)  the  process  and  engineering  activities  by  which  systems  are  constructed,  evolved,  and  sustained, 
including  mechanisms  for  measurement  of  critical  attributes  and  for  management  of  alternatives  and 
commitments,  (3)  the  supporting  models  and  techniques  through  which  evidence  can  be  created  to 
support  assurance  judgments.  The  SERC's  focus  is  on  the  last  of  these  three  -  achieving  high  levels  of 
trustworthiness  -  but  it  is  recognized  that  a  strategic  approach  is  required  that  builds  on  the  interplay  of 
these  three  critical  dimensions  of  consideration.  This  interplay,  evident  in  the  diagnosis  of  assurance 
failures  and  root  cause  analyses,  defines  the  scope  of  this  grand  challenge. 

The  strategy  taken  in  this  area  has  four  principal  features:  (1)  the  expression,  retention,  and  analysis  of 
diverse  kinds  of  information  related  to  requirements,  design,  implementation,  and  operation;  (2) 
mechanisms  whereby  the  potential  consequences  of  decisions  and  engineering  commitments  can  be 
understood  as  early  as  possible  in  the  process,  including  approaches  such  as  iteration,  prototyping, 
modeling  and  simulation,  analytic  methods,  and  other  approaches;  (3)  support  for  these  practices 
(information  management  and  tight  feedback)  across  the  entire  lifecycle  starting  from  the  earlier  stages 
of  requirements  formulation  and  encompassing  architecture,  design,  implementation,  evaluation, 
integration  into  operating  environments  and  ecosystems,  and  operation;  and  (4)  ability  to  respond 
effectively  to  changes  in  the  mission  operating  environment,  the  SoS  context,  and  the  infrastructure 
environment. 

These  four  features  give  rise  to  the  seven  areas  of  technical  emphasis  identified  in  the  Systemic  Assurance 
Program.  Initial  emphasis  to  address  the  TS  Research  Grand  Challenge  is  on  establishing  a  baseline  for 
assurance  standards  and  practice,  with  emphasis  on  developing  a  framework  for  the  evaluation  of 
practices  and  techniques.  This  framework  consists  of  a  set  of  meta-criteria  -  criteria  through  which 
evaluation  practices  (including  practice-specific  criteria  for  process  and  structure)  can  themselves  be 
assessed  and  compared.  Importantly,  the  meta-criteria  also  enable  assessment  of  the  potential  impact  of 
emerging  practices,  models,  tools,  and  techniques.  A  candidate  inventory  of  meta-criteria  has  been 
identified  and  in  future  work  will  be  refined  with  the  development  of  scales  and  value  weightings. 
Additionally,  considerable  technical  progress  has  been  made  on  techniques  for  design,  modeling,  and 
analysis,  focusing  at  three  crucial  stages  of  SE:  requirements,  architecture  and  design,  and 
implementation. 
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This  initial  effort  will  enable  progress  on  aspects  of  technology  and  practice  that  relate  to  the 
representation  and  management  of  diverse  kinds  of  evidence,  including  implementation  artifacts,  models 
and  analysis  results,  and  dependency  and  traceability  linkages  among  these  things.  This  creates  the 
foundation  for  an  explicitly  evidence-based  approach  that  offers  a  pathway  for  incremental  improvement 
of  capability  along  with  a  framework  of  meta-criteria  for  evaluation  of  progress. 


5.2.2  Strategies  to  Address  the  Trusted  System  Grand  Challenge 

Successfully  executing  the  following  strategies  will  make  significant  progress  towards  addressing  the  TS 
Grand  Challenge: 

1.  Design  for  System  Assurance  and  Trust:  Develop  design  patterns  and  systems  architectures,  with 
corresponding  systems  engineering  principles  guiding  application,  and  associated  design  analysis 
MPTs  for  early  assurance  of  needed  properties 

2.  Understand  the  Cost  of  Assurance  and  Ensure  Cost-Effective  Assurance:  Develop  MPTs  that  enable 
understanding,  predicting,  and  ensuring  the  cost-effectiveness  of  implementing  high-assurance 
policies  and  requirements,  especially  on  complex  systems  and  complex  systems  of  systems 

3.  Understand  and  Ensure  Balanced  Tradeoffs  Between  Assurance  "ilities"  and  Other  "llities": 
Develop  MPTs  that  enable  understanding,  predicting,  and  ensuring  cost-effective  relationships 
among  assurance  policies/requirements  and  other  "ilities",  such  as  usability,  interoperability,  and 
maintainability 

4.  Measure  System  Assurance:  Develop  MPTs  that  allow  measuring  "how  much"  assurance  of 
needed  properties  a  system  has,  and  that  permit  comparison  of  the  relative  assurance  and  trust 
provided  by  alternative  systems 

Recall  that,  for  the  SERC,  development  of  an  MPT  includes  validation  and  transition. 

Two  research  programs  directly  implement  these  four  strategies: 

•  Systemic  Security.  The  most  compelling  need  for  assurance  of  trust  is  in  the  area  of  system 
security.  Given  the  numerous  sources  of  security  breaches  available  at  low  cost  to  attackers,  a 
major  concern  is  to  make  DoD  systems,  SoSs,  and  enterprises  harder  to  attack,  while 
simultaneously  making  them  more  difficult  and  expensive  to  penetrate  and  damage. 

•  Systemic  Assurance.  Besides  security  attacks,  there  are  numerous  sources  of  system  disruption 
such  as  natural  disasters,  system  misuse,  system  overload,  system  component  wear  out,  and 
defects  in  a  system's  requirements,  design,  or  construction.  Preventing  or  otherwise  addressing 
these  disruptions,  which  cause  loss  of  stakeholders'  lives,  health,  capability,  property,  or  financial 
assets,  require  significant  improvements  in  trust  not  only  for  current  systems,  but  for  the  more 
complex  and  dynamic  DoD  systems,  SoSs,  and  enterprises  of  the  future. 

In  addition,  improvements  in  system  trust  have  been  and  are  being  addressed  in  the  other  SERC  research 
areas,  particularly  in  SEMT  and  its  current  projects:  System  Qualities,  Interactive  Model-Centric  SE,  and 
Quantitative  Technical  Risk.  Example  contributions  from  these  and  earlier  SEMT  projects  include  SERC 
insights  such  as  those  from  projects  addressing  technical,  integration  and  manufacturing  maturity  level 
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assessment,  risk  management  precepts,  the  enterprise  management  approach  to  quantifying  early-SE 
risks,  the  MIT  epoch-era  approach  to  assurance  under  uncertainty,  and  the  set-based  vs.  point-design 
approach  to  assurance  of  systems  undergoing  continuing  and  extensive  change.  The  synergies  among 
these  research  projects  will  be  addressed  and  enhanced  by  periodic  cross-research-area  workshops. 


5.2.3  Systemic  Security  Program 

The  goal  of  the  Systemic  Security  Program  is  to  develop  MPTs  that  enable  safe,  secure,  dependable 
defense  systems  that  are  resilient  to  cyber  and  other  threats  through  systemic  security  approaches  that 
complement  current,  incomplete  perimeter/network.  This  goal  is  being  achieved  by  reversing  cyber 
security  asymmetry  from  favoring  our  adversaries  (small  investment  in  straightforward  cyber  exploits 
upsetting  major  system  capabilities),  to  favoring  the  US  (small  investments  for  protecting  the  most  critical 
system  functions  using  systems-aware  cyber  security  solutions  that  require  very  complex  and  high  cost 
exploits  to  defeat).  Building  on  the  four  TS  Area  strategies  to  address  the  TS  Grand  Challenge,  the  Systemic 
Security  strategies  are: 

1.  Design  for  System  Security.  Develop  MPTs  that  develop  solution  selections  on  a  mission  security 
basis  as  opposed  to  a  subsystem  basis,  recognizing  that  the  interaction  between  subsystems 
provides  opportunities  for  adversaries  regarding  cyberattacks,  and  also  provides  potential 
economies  for  defenders  regarding  identification  of  the  most  cost  effective  way  for  achieving 
mission  security. 

2.  Develop  design  patterns  and  security  architectures  that  enable  security  to  be  based  on  the  specific 
properties  of  the  system  and  its  implementation  as  a  complement  to  traditional  perimeter 
strategies:  Address  security  of  weapon  systems,  sensor  systems,  physical  plant  systems  as  well  as 
IT  systems  within  the  context  of  SoSs  applied  to  military  missions;  e.g.,  air  defense,  point  target 
defense,  and  warning  systems.  Account  for  operational  procedures  and  human  factors  in  the  SoS 
context. 

3.  Support  security  requirements  assessments  that  directly  address  cost  and  achievement  of  cost- 
effective  security:  Develop  MPTs  that  enable  understanding,  predicting,  and  ensuring  the  cost- 
effectiveness  of  implementing  specific  security  policies  and  requirements,  especially  on  complex 
systems  and  complex  systems  of  systems 

4.  Explore  overlaps  and  difference  in  security  monitoring:  Initiate  exploration  efforts  that  identify  the 
overlaps  and  differences  between  security  monitoring  as  employed  in  the  Systems-Aware  concept 
and  performance  monitoring  for  autonomy,  recognizing  that  autonomous  systems  will  need  to 
include  monitoring  functions  for  performance  assurance. 

This  research  program  implements  all  four  TS  strategies  above.  Table  5.2-1  offers  a  description  of  the 
single  long-term  project  currently  underway  in  this  program  and  which  strategies  that  project  primarily 
supports. 
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Table  5.2-1.  Projects  in  the  Systemic  Security  Program 


Project 

Started 

Purpose 

Primary  TS 
Supported  Strategies 

Systems-Aware 

Security 

2011 

Develop  and  then  refine  Systems-Aware  MPTs 
and  pilot  them  in  multiple  application  areas 

1,  2,  3,  4 

5. 2. 3.1  Systems-Aware  Security  Project 

In  2011,  SERC  RT-28:  Systems-Aware  Security  developed  a  rapid  prototype  security  capability  that  includes 
(1)  data  continuity  checking  within  the  application,  (2)  real-time  virtual  configuration  hopping  of  selected 
command  and  control  functions  across  multiple  operating  systems  to  provide  defense  through  diversity, 
(3)  real-time  physical  configuration  hopping  to  both  provide  defense  through  diversity  and  resilience  in 
the  face  of  successful  attacks,  and  (4)  a  closed  loop  control  system  for  automatic  restoration  from  a 
successful  attack.  In  2012-2013,  SERC  RT-42:  Security  Engineering  Pilot  developed  a  prototype  flight- 
capable  security  capability  directed  toward  an  unmanned  air  vehicle  (Outlaw  aircraft  containing  an 
embedded  Piccolo  flight  control  system)  carrying  a  pre-existing  set  of  surveillance  equipment 
(video/infrared  cameras,  radar,  and  a  signals  intelligence  package). 

The  original  focus  of  the  project  has  concluded.  It  served  to  provide  a  greater  understanding  and  visibility 
for  the  Systems-Aware  concept  and  it  identified  the  need  and  opportunity  for  addressing  mission-based 
security  and  human  factors  from  an  SoS  perspective.  Subsequent  activities  will  involve  the  tailoring  of  the 
capabilities  to  other  domains,  and  associated  evaluation  and  refinement,  along  with  monitoring  and 
refinement  of  existing  fielded  capabilities. 

Table  5.2-2  shows  the  focus  and  deliverables  in  the  project  through  2016.  Continuing  extensions  and 
upgrades  will  be  pursued  in  2017  and  2018. 


Table  5.2-2.  Systems-Aware  Security  Project  Timeline* 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Concept  definition,  prototyping,  piloting 

Initial  Systems-Aware  Security  capability 

2014 

Extended  evaluation,  refinement,  and 

Packaging 

Tailorable  Systems-Aware  Security  capability 

2015 

Mission  level  security  concept  development 

SoS  Systems-Aware  Security  capability  concept 
and  prototype,  tool-supported  methodology  for 
development  security  requirements,  exploration 
of  human  factors,  rapid  security  prototype  for  an 
existing  military  system  (AIMES) 

2016 

Through  concept  refinement  efforts  and 
prototyping  efforts,  direct  application  of  2015 
results  toward  application  into  existing 
military  systems 

Two  new  domain  Systems-Aware  Security 
capabilities 
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The  Systems-Aware  Security  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.2-3 
and  5.2-4  below. 


Table  5.2-3.  Systems-Aware  Security  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Initiated  collaboration  with  Navy  10th  Fleet  (Cyber  Command)  and  JHUAPL  in 
addressing  requirements  methodology  and  support  tools,  including  organizing  a 
workshop  to  introduce  the  research  to  them.  Met  with  large  Aegis  Program 
Manager  group  to  engage  their  interest. 

•  Plan  Early 

•  Engage  Community 

2 

The  University  of  Virginia  has  licensed  Systems-Aware  technology  to  a  start-up 
company  (MSI)  engaging  in  offering  new  security  products  and  services  related 
to  Systems-Aware  concept  and  have  initiated  efforts  to  gain  new  patents. 

•  Engage  Community 

•  Pilot  Continuously 

•  Productize 

3 

Integrated  AIMES  prototype  into  a  live  prototype  SoS  environment  to  highlight 
mission-oriented  approach  to  security  including  an  operational  system. 

•  Engage  Community 

•  Pilot  Continuously 

•  Productize 

4 

Initiated  projects  with  NIST  on  3D  Printers  and  Virginia  State  Police  focused  on 
automobiles  that  have  both  provided  confirmation  of  potential  value  and 
provided  new  elements  of  learning  for  transition  into  military  systems. 

•  Engage  Community 

•  Pilot  Continuously 

•  Productize 

5 

Include  Air  Force  Institute  of  Technology  as  part  of  the  Fluman  Factors  research 

efforts. 

•  Engage  Community 

6 

Involve  the  DoD  Chief  Information  Officer  in  the  definition  of  the  Cloud 
computing  portion  of  the  project. 

•  Plan  Early 

•  Engage  Community 

7 

Involve  the  Deputy  Assistant  Secretary  of  Defense  for  Emerging  Capability  and 
Prototyping)  in  supporting  the  definition  of  a  rapid  prototyping  project  that  is 
directed  toward  development  of  an  operational  prototype  radar  system  with 
System  Aware  security  capabilities. 

•  Plan  Early 

•  Engage  Community 

Table  5.2-4.  Systems-Aware  Security  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Application  to  existing  Army/Air  Force  AIMES  system,  etc. 

Progress 

(approval, 

adoption) 

•  The  project  has  published  5  journal  articles  with  one  currently  under  review,  4  conference 
papers,  4  technical  reports,  and  numerous  public  presentations. 

•  Air  Force,  Navy,  DoD  Chief  Information  Officer  are  engaged  in  project  efforts;  MITRE, 
JFIUAPL  have  provided  support  to  the  project 

•  Through  the  Virginia  Cybersecurity  Commission,  have  initiated  an  economic  development 
plan  that  addresses  support  for  education  and  research  activities  that  bring  together  the 
cyber-physical  systems  community  with  the  cybersecurity  community 
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5.2.4  Systemic  Assurance  Program 

Besides  security,  the  engineering  of  resilient  DoD  systems  requires  assurance  of  safety,  reliability, 
availability,  durability,  survivability,  maintainability,  evolvability,  adaptability,  and  sustainability.  Systems 
cannot  be  deployed  until  customer  organizations  judge  them  fit  for  use  in  the  mission  environment.  These 
assurance  judgments  must  be  based  on  evidence  that  a  system  manifests  not  just  the  necessary 
functionality  but  also  these  quality  attributes,  and  at  a  level  appropriate  to  the  operating  environment. 
All  of  this  assurance  needs  to  be  achieved  for  increasingly  complex,  dynamic,  cyber-physical-human  net- 
centric  systems,  SoSs  and  enterprises  with  needs  for  rapid  response  incompatible  with  most  heavyweight 
assurance  MPTs.  Carnegie  Mellon  University  (CMU)  has  been  a  leader  in  developing  assurance  MPTs  in 
such  areas  as  architectural  style  analysis,  race  and  deadlock  detection  for  multicore  and  other  concurrent 
systems,  appropriate  test  case  generation,  model  checking,  and  assurance-case  analysis.  These 
techniques  must  be  applied  not  just  in  anomaly  detection,  but  also  leading  to  stronger  possibilities  for 
positive  assurance  and  to  greater  scalability  to  large  systems  and  to  more  rapid  execution.  These 
techniques  have  been  successfully  applied  in  such  areas  as  the  High  Level  Architecture  analysis  for 
networked  DoD  models  and  simulations,  cyber-physical  robotic  systems,  and  extremely  large  commercial 
Java  programs. 

The  Systemic  Assurance  Program  currently  has  one  project,  of  the  same  name,  that  builds  on  these 
capabilities  and  data  sources  to  improve  both  the  level  of  assurance  obtainable  and  the  cost-effectiveness 
of  assurance-related  effort,  with  a  goal  of  rapid  recertification  to  support  evolving  components  and 
systems. 

Table  5.2-5  below  offers  a  description  of  the  sole  project  currently  in  this  program. 

Table  5.2-5.  Projects  in  th e  Systemic  Assurance  Program 


Project 

Started 

Purpose 

Primary  TS 
Supported  Strategies 

Systemic 

Assurance 

2013 

Develop  MPTs  that  combine  technical  analysis  of  system 
artifacts  and  requirements  and  architecture  techniques 
to  promote  assurance  and  resiliency 

1,  2,  3,  4 

5. 2.4.1  Systemic  Assurance  Project 

The  Systemic  Assurance  Project  combines  technical  analysis  of  system  artifacts  and  requirements  and 
architecture  techniques  to  promote  assurance  and  resiliency.  An  important  goal  is  to  develop 
incrementally  composable  combinations  of  MPTs  and  data  based  composition  guidance  for  obtaining  the 
most  cost  and  schedule-effective  combinations  for  the  assurance  of  necessary  system  properties. 
Particular  areas  of  emphasis  will  be  exploiting  analogies  with  successful  techniques  in  other  domains,  such 
as  building  codes;  better  management  of  chains  of  evidence  to  support  ongoing  re-evaluation  for  rapidly 
evolving  systems  both  in  development  and  in  sustainment/modernization;  enhancing  MPTs  with  language 
extensions  for  assurance  assertions  or  context  metadata;  support  for  dynamic  adaptiveness  and  resiliency 
in  architectural  design;  and  data  management  and  metrics  for  more  evidence-based  design,  development, 
and  decision  support. 
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Reflecting  strong  natural  ties  between  the  Systemic  Assurance  Project  and  the  ilities  Tradespace  and 
Affordability  Systems  Qualities  Project  (SQ)  described  in  5. 3. 3.1,  extensive  coordination  will  be  pursued 
between  the  two  projects. 

The  Systemic  Assurance  Project  implements  all  four  TS  Area  strategies  above.  It  includes  seven  research 
and  technology  subprojects  led  by  senior  CMU  researchers,  with  extensive  internal  coordination 
mechanisms  to  exploit  the  synergies  among  the  various  technical  approaches.  The  CMU  team  members 
have  an  extensive  network  of  collaborations  and  partnerships  with  DoD  activities,  other  government 
agencies,  the  Software  Engineering  Institute,  other  SERC  universities,  and  numerous  industrial  firms. 

The  Systemic  Assurance  Project  has  four  technical  themes  related  to  advancing  assurance  capability: 

•  Evidence  and  traceability.  Facilitate  early  validation  by  accumulating  assurance-related  evidence 
and  creating  traceability  structures  during  development. 

•  Requirements,  architecture,  composition,  variabilities.  Address  assurance  goals  in  the  earliest 
phases  of  development.  Enable  composition  of  assurance  judgments  for  components  into  overall 
judgments  for  systems. 

•  Direct  analysis.  Use  semantics-based  techniques  to  enhance  confidence  and  scalability,  focusing 
on  challenges  significant  for  modern  systems,  such  as:  framework  protocol  compliance,  highly 
versioned  systems,  automatic  defect  repair,  safe  concurrency,  and  other  areas. 

•  Combined  methods.  Integrate  multiple  methods  to  evaluate  quality  attribute  requirements  for 
heterogeneous  systems,  combining  informal  and  formal,  static  and  dynamic,  and  development 
and  operational  monitoring. 

The  research  team  addresses  these  themes  through  seven  subprojects: 

1.  Practice  baselining.  Develop  baseline  and  intervention  models  for  a  selection  of  current  standards 
and  practices  (identified  in  collaboration  with  DoD  stakeholders),  refining  technical  understanding 
of  gaps  and  limitations.  This  baselining  effort  is  essential  to  support  a  measurement-based 
approach  to  documenting  the  impact  of  the  proposed  new  technologies  and  process 
interventions.  This  includes  identifying  the  key  criteria  and  dimensions  of  measurement. 

2.  Evidence  building.  Undertake  engineering  design  effort  focused  on  integrating  improved 
capability  for  traceability  and  other  features  required  to  support  explicit  modeling  and 
management  of  chains  of  evidence.  A  key  focus  is  to  demonstrate  that  it  is  possible  to  enhance 
existing  tools  and  environments,  including  both  integrated  development  environments  and  team 
tools,  with  relatively  little  disruption  to  established  team  practices  and  metrics. 

3.  Recertification  practice.  Design  and  implement  experiments  to  address  the  challenge  of  rapid 
recertification.  These  include  capturing  evidence  and  assurance-related  reasoning  (assurance 
cases,  models,  analyses,  configuration  management,  etc.).  This  area  of  rapid  recertification  is 
critical  to  iterative,  incremental,  and  staged  development  practices.  It  is  also  critical  to  systems 
with  supply  chains  that  include  externally  developed  components  and  infrastructure  such  as 
commercial  and  open-sourced  databases,  operating  systems,  frameworks,  and  libraries.  (Almost 
all  larger-scale  software-reliant  systems  have  this  characteristic.) 

4.  Architecture  assessment.  Develop  a  framework  for  assessment  of  architecture-derived  quality 
attributes,  focusing  on  architectural  modeling  and  the  relationship  of  architectural  and 
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compositional  models  with  quality  outcomes.  This  is  essential  in  order  to  ensure  that  key 
decisions  made  at  early  lifecycle  phases  will  have  intended  quality  outcomes. 

5.  Requirements  and  quality  validation.  Develop  requirements  elicitation  and  management 
approaches  that  better  address  quality  and  policy  objectives.  Requirements  elicitation  and 
management  is  one  of  the  earliest  areas  of  focus  in  an  engineering  process,  and  decisions  at  this 
point  can  have  tremendous  leverage  on  quality  outcomes.  This  work  is  directed  at  providing  more 
immediate  assessments  of  the  potential  outcomes  of  early  requirements-related  decisions.  By 
improving  models,  it  becomes  possible  to  better  manage  the  linkage  of  requirements  and 
architectural  decisions. 

6.  Technical  quality  attributes.  Augment  and  collaborate  with  diverse  existing  efforts  focused  on 
technical  means  to  address  particular  quality  criteria.  Many  of  these  quality  criteria  are  emerging 
as  significant  challenges  because  they  tend  to  defy  conventional  testing  and  inspection 
techniques.  These  include,  for  example,  a  number  of  attributes  related  to  safe  concurrency, 
compliance  with  application  program  interface  rules-of-the-road,  cyber-physical  architectural 
compliance,  state  and  access  management  for  shared  objects,  taint  and  flow  and  other  security- 
related  attributes,  and  others. 

7.  Tools,  automation,  and  usability.  Identify  and  advance  areas  in  support  of  increasing  automation, 
in  order  to  reduce  workload  of  developers  and  evaluators  and  to  advance  existing  workload 
forward  in  the  process,  with  immediate  rewards.  The  purpose  of  this  is  to  frame  an  ultimately 
more  quantitative  business  case  for  adoption  based  on  increased  return  on  investment  for 
assurance-related  effort  and  reduced  uncertainty  (lesser  variance  in  estimate  "cones").  This  is 
supportive  of  the  longer-term  goal  of  a  "positive  benefit"  model  for  the  adoption  of  assurance- 
related  practices.  It  also  supports  a  stakeholder-engaged  process  model  analogous  to  building 
codes. 


Figure  5.2-1.  Systemic  Assurance  Principal  Task  Interdependencies 
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There  are  important  synergies  and  interactions  among  these  seven  subprojects,  with  the  principal 
features  outlined  in  Figure  5.2-1  above  (subproject  numbers  are  in  brackets). 

Table  5.2-6  shows  the  focus  and  deliverables  in  the  Systemic  Assurance  Project. 


Table  5.2-6.  Systemic  Assurance  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Startup 

Prepare  for  and  hold  kickoff  meeting  with  sponsor,  Systemic 

Security,  and  System  Qualities  representatives 

2014 

Initiate  efforts  in 
subprojects  1,  4,  5,  and  6. 

Identify  principal  candidate  software  code  bases  and  cyber-physical 
systems  for  empirical  studies.  Candidates  include  commercial  and 
government  partner  systems  and  significant  open  source  systems 
of  government  interest.  Begin  empirical  studies  and  solution 
explorations. 

2015 

Continue  effort  on 
subprojects  1,  4,  5,  and  6. 
Identify  most  promising 
solution  options,  begin 
solution  research  and 
development.  Initiate  effort 
on  subprojects  2  and  7. 

Complete  initial  baseline  analysis  and  identify  feature  points  to 
address  in  framing  potential  revised  practices.  Identify  technical 
blockers  and  potential  remediations. 

Initiate  work  on  tool  design,  building  on  capabilities  of  established 
integrated  development  environment  and  team  tools.  Develop  a 
"minimal  perturbation"  model  that  augments  the  tooling  in  specific 
ways  to  address  the  traceability,  modeling,  and  analysis  challenges 
of  themes  2,  3,  and  7. 

Continue  work  on  specific  technical  attributes,  focusing  on  the 
challenges  of  attributes  that  tend  to  defy  conventional  testing  and 
inspection  (themes  4,  5,  and  6). 

Develop  assessment  techniques  to  address  the  requirements 
elicitation  goals  of  theme  5. 

2016 

Elaborate  and  mature 
solutions;  engage  with 
stakeholders;  conduct  trial 
deployments.  Identify  and 
explore  new  areas  of  high- 
potential  research 

Engage  with  stakeholders  (potentially  in  collaboration  with  SEI)  to 
continue  the  baselining  and  criteria  definition  of  theme  1,  leading 
to  a  preliminary  formulation  of  an  alternative  model  based  on 
building  code  ideas  (themes  2,  3,  and  7). 

Advance  efforts  in  technical  and  tooling  thrusts,  producing 
exemplar  evidence-based  assurance  data  for  existing  major 
components. 

Conduct  trial  deployments  of  advanced  tooling  and  metrics 
capabilities  with  professional  development  teams  in  partner 
organizations. 

Advance  requirements  and  architecture  efforts,  identifying 
candidate  "emerging  best  practices"  to  support  architecture-led 
iterative  development  efforts. 
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Year 

Focus 

Key  Deliverables 

Demonstrate  rapid  recertification  for  one  or  more  of  the  exemplar 

Demonstrate  and  interact 

systems  for  which  evidence  is  created  in  2016. 

2017 

with  baseline  capability 
users;  extend  and  apply 
additional  solutions. 

Develop  evidence-based  approaches  for  dynamic  and  resilient 
systems  potentially  with  shape-shifting  architectures. 

Mature  new  areas  of  high- 
potential  research 

Engage  with  stakeholders  to  advance  experimental  concepts  for 
new  evidence-based  approaches  to  "designed-in  assurance 
support"  for  larger-scale  component-based  systems. 

Advance  the  traceability  capabilities  in  tooling  to  support  a  concept 
of  continuous  re-evaluation  and  reconstruction  of  evidence  to 

Based  on  experience  with 
existing  results,  identify  and 

support  a  model  of  "continuous  re-certification." 

2018 

pursue  further  baseline 

Enhance  tool  prototypes  to  include  broader  ranges  of  critical 

extensions  and  new-idea 

technical  quality  attributes  (theme  6). 

projects 

Identify  advances  in  modeling,  language,  and  analysis  to  enable 
broad  adoption  of  evidence-based  approaches. 

The  Systemic  Assurance  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.2-7  and 
5.2-8  below. 


Table  5.2-7.  Systemic  Assurance  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Develop  and  apply  technical  analytic  approaches  to  extant  large-scale  software 
corpora  reflective  of  the  kinds  of  software  components  in  large-scale  DoD  and 

1C  systems.  Publish  technical  assessment  of  the  effectiveness  of  the  models 
and  analyses. 

•  Plan  Early 

2 

Consult  acquisition  and  sustainment  subject-matter  experts  regarding  the 
content  of  the  proposed  meta-criteria  in  order  to  assess  the  significance  of  the 
individual  meta-criteria  and  validate  and  prioritize  them.  This  consultation 
could  include  both  individual  subject  matter  experts  and  possibly  also  a 
workshop. 

•  Long  and  Short 

Term 

•  Engage  Community 

3 

Consult  experts  associated  with  the  identified  baseline  practices  (sub-project 

1)  to  refine  and  validate  the  team's  assessment  with  respect  to  the  meta¬ 
criteria.  Identify  challenges  and  successes  for  the  individual  practices. 

•  Plan  Early 

•  Engage  Community 

4 

Work  with  requirements  and  architecture  subject  matter  experts  to  assess 
particular  challenges  faced  in  developing  these  high  level  models  and,  in 
particular,  managing  traceability  between  these  models  and  other 
development  artifacts. 

•  Engage  Community 

•  Pilot  Continuously 
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Table  5.2-8.  Systemic  Assurance  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

•  Direct  assessment  of  representative  current  practice  guidelines,  with  input  from  OSD 

(relevance, 

regarding  the  selection  of  representative  practice  guidelines 

practicality) 

•  Extensive  technical  publication  related  to  sub-projects  4,  5,  6,  and  7 

Progress 

(approval, 

adoption) 

•  Incorporation  of  results  of  all  thrusts  into  academic  curricula 

•  Exposure  and  validation  of  meta-criteria  with  DoD  stakeholders. 

5.2.5  Other  SERC  Non-Core  Funded  Projects 

During  the  time  of  the  Technical  Plan,  the  SERC  has  executed  one  non-Core-funded  TS  Area  project  as 
briefly  described  in  Table  5.2-9  below. 


Table  5.2-9.  TS  Area  Non-Core  Funded  Projects 


Project 

Sponsor 

Description 

Software 

Reliability 

Modeling 

Navy 

The  three  primary  tasks  are  to  (1)  Design  and  implement  a  software  tool 
framework  that  enables  the  automatic  application  of  software  reliability  models; 

(2)  Analyze  the  software  testing  and  reliability  policies  and  processes  within  DoD  to 
identify  suitable  models  to  optimize  their  efficiency  while  ensuring  software 
produced  is  both  reliable  and  cost  effective,  and  (3)  Conduct  research  on  statistical 
algorithms  that  will  ensure  the  robustness  of  software  reliability  models,  including 
methods  based  on  the  Expectation  Maximization  algorithm,  Bayesian  Modeling, 
and  the  Maximum  Entropy  Principle. 

5.3  Systems  Engineering  and  Systems  Management  Transformation  (SEMT) 

Traditional  DoD  systems  engineering  and  management  (SE&M)  practices  have  focused  on  the  definition 
and  acquisition  of  individual  standalone  platforms  within  a  relatively  stable  environment.  They  have 
generally  used  slow,  sequential  processes  that  often  commit  to  requirements  before  their  development 
implications  are  fully  understood.  This  has  caused  much  late  and  expensive  rework,  along  with  brittle, 
hard-to-modify  architectures. 

However,  DoD's  current  and  future  environment  requires  that  such  platforms  and  their  human  and 
software  elements  be  defined,  integrated,  and  evolved  within  highly  complex  and  dynamic  net-centric 
systems  of  systems  and  enterprises.  The  need  for  more  innovative,  life-cycle-affordable  systems 
engineering  and  management  approaches  has  been  recognized  by  the  current  DoD  leadership  in  such 
initiatives  as  Better  Buying  Power  and  the  proposed  life-cycle  strategy  for  the  DoD  Next-Generation 
Bomber.  This  requires  research  focused  on  transforming  traditional  SE&M  MPTs  to  meet  these  current 
and  future  DoD  mission  needs. 

Goal:  Transform  systems  engineering  and  its  associated  management  approaches  away 
from  systems  designed  for  optimal  performance  against  a  static,  pre-specified  set  of 
requirements  over  long  procurement  cycles  to  approaches  that  enhance  the  productivity 
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of  engineers  to  rapidly  and  concurrently  develop  cost-effective,  flexible,  agile  systems 
that  can  respond  to  evolving  threats  and  mission  needs.  "Systems"  covers  the  full  range 
of  DoD  systems  of  interest  from  components  such  as  sensors  and  effectors  to  DoD-wide 
net-centric  systems  of  systems  and  enterprises.  "Effectiveness"  covers  the  full  range  of 
needed  system  quality  attributes  or  ilities,  such  as  reliability,  availability,  maintainability, 
safety,  security,  performance,  usability,  scalability,  interoperability,  speed,  versatility, 
flexibility,  and  adaptability,  along  with  composite  attributes  such  as  resilience, 
suitability,  and  sustainability  to  support  the  desired  mission  performance.  "Cost"  covers 
the  full  range  of  needed  resources,  including  present  and  future  dollars,  calendar  time, 
critical  skills,  and  critical  material  resources. 


5.3.1  SEMT  Grand  Challenge  and  Current  Progress 

The  SEMT  grand  challenge  to  achieve  the  SEMT  goal  is  to: 

Move  the  DoD  community's  current  systems  engineering  and  management  MPTs  and  practices 
away  from  sequential,  document-driven,  hardware-centric,  point-solution,  acquisition-oriented 
approaches;  toward  concurrent,  portfolio  and  enterprise-oriented,  hardware-software-human 
engineered,  model-driven,  set-based,  full  life  cycle  approaches.  These  will  enable  much  more 
rapid,  flexible,  scalable  definition,  development  and  deployment  of  the  increasingly  complex, 
cyber-physical-human  DoD  systems,  systems  of  systems  and  enterprises  of  the  future. 

In  keeping  with  its  central  position  among  the  four  SERC  thematic  research  areas  in  Figure  3-1,  the  SEMT 
research  area  includes  collaborative  efforts  with  the  other  three  SERC  research  areas  with  respect  to  their 
Grand  Challenges.  This  collaboration  provides  SEMT  with  greater  understanding  of  how  its  research 
efforts  can  help  address  their  Grand  Challenges,  and  provides  SEMT  with  insights  on  how  its  research 
results  can  span  multiple  Grand  Challenges. 

An  example  of  such  collaboration  is  SEMT's  research  support  of  the  ESOS  Area  Grand  Challenge  of  creating 
SE  principles  and  MPTs  for  SoS  SE  that  generate  affordable  and  overwhelming  competitive  advantage  over 
its  current  and  future  adversaries.  SEMT's  research  results  in  cost  estimation  of  SoS  SE  effort,  combined 
with  its  results  in  SysML  parametric  architecture  modeling  and  previous  research  efforts  in  the 
Requirements  for  Net-Centric  SE  Project  conducted  before  the  start  of  this  Technical  Plan,  have  been 
integrated  to  provide  SoS  SE  cost  estimation  capabilities  for  affordable  SoSs. 

A  further  example  involves  SEMT's  research  support  of  the  TS  Area  Grand  Challenge  of  achieving  much 
higher  levels  of  system  trust  for  the  increasingly  complex,  dynamic,  cyber-physical-human  net-centric 
systems  and  systems  of  systems  of  the  future.  A  workshop  involving  the  TS  Systems  Assurance  Project 
and  the  SEMT  Systems  Qualities  Tradespace  and  Affordability  Project  has  led  to  collaborative  efforts  in 
identifying  and  quantifying  the  synergies  and  conflicts  among  strategies  for  assuring  security  and  safety 
qualities  and  strategies  for  achieving  affordability,  flexibility  and  mission  assurance  qualities. 

Additional  examples  involve  SEMT's  research  support  of  the  HCD  Area  Grand  Challenge  of  dramatically 
accelerating  the  professional  development  of  highly  capable  systems  engineers  and  technical  leaders  in 
DoD  and  the  defense  industrial  base.  One  example  cited  above  is  SEMT's  simulation-based  research  on 
using  Agile-Lean-Kanban  methods  to  help  DoD  and  industry  SEs  prioritize  and  accelerate  rapid  DoD 
responses  to  new  threats  and  opportunities.  Other  examples  include  the  previous  SEMT  Graphical 
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ConOps  Project  research  and  the  current  Interactive  Model-Centric  SE  Project,  both  focused  on  how  to 
better  support  human  visualization  and  decision  support  in  defining  and  developing  complex  cyber- 
physical-human  systems. 

SEMT  has  combined  insights  from  these  collaborations  and  support  of  its  OSD,  Air  Force,  Army,  Navy  and 
DoD  Agency  research  sponsors  to  formulate  and  create  stronger  SE  and  management  foundations  for 
addressing  the  SEMT  Grand  Challenges  above.  These  foundations  include  set-based  design  of  DoD 
systems,  quantifying  system  qualities  and  system  risks,  an  ontology  for  clarifying  the  complexities  of 
system  qualities  and  their  interactions,  and  methods  for  evidence  and  risk-based  decision  support  for 
evolutionary,  concurrent  SE  and  system  development  projects.  All  of  these  efforts  continue  to  evolve  and 
identify  further  challenges,  as  described  in  the  future  plans  below. 


5.3.2  Strategy  to  Address  the  SEMT  Grand  Challenge 

Successfully  executing  the  following  strategies  will  make  significant  progress  towards  addressing  the 
SEMT  Grand  Challenge: 

1.  Make  Smart  Trades  Quickly:  Develop  MPTs  to  enable  stakeholders  to  be  able  to  understand  and 
visualize  the  tradespace  and  make  smart  decisions  quickly  that  take  into  account  how  the  many 
characteristics  and  functions  of  systems  impact  each  other 

2.  Rapidly  Conceive  of  Systems:  Develop  MPTs  that  allow  multi-discipline  stakeholders  to  quickly 
develop  alternative  system  concepts  and  evaluate  them  for  their  effectiveness  and  practicality 

3.  Balance  Agility  and  Assurance:  Develop  SE  MPTs  that  work  with  high  assurance  in  the  face  of  high 
uncertainty  and  rapid  change  in  mission,  requirements,  technology,  threats,  and  other  factors  to 
allow  a  system  to  be  rapidly  acquired  and  responsive  to  both  anticipated  and  unanticipated 
changes  in  the  field 

4.  Align  with  Engineered  Resilient  Systems:  Align  research  to  both  leverage  the  research  and 
technology  results  of  the  Engineered  Resilient  Systems  (ERS)  program,  and  contribute  to  it;  e.g., 
ERS  efforts  to  define  new  approaches  to  tradespace. 

Recall  that,  for  the  SERC,  development  of  an  MPT  includes  validation  and  transition. 

Four  current  Core-funded  SERC  research  programs  have  been  implementing  these  strategies: 

•  Affordability  and  Value  in  Systems 

•  Quantitative  Risk 

•  Interactive  Model-Centric  Systems  Engineering 

•  Agile  Systems  Engineering 

In  addition,  SEMT  has  been  successful  in  attracting  complementary  funding  from  the  Air  Force  Space  and 
Missile  Command,  the  Army  Engineer  Research  and  Development  Center,  several  Navy  organizations,  the 
Marine  Corps,  and  the  National  Science  Foundation,  all  of  which  extend  and  experimentally  apply  the 
capabilities  developed  under  Core  funding. 
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5.3.3  Affordability  and  Value  in  Systems  Program 

The  Affordability  and  Value  in  Systems  Program  has  been  underway  since  2014.  It  will  continue  to  focus 
on  improving  DoD's  ability  to  deal  with  System  Qualities  (SQs),  also  known  as  Non-Functional 
Requirements  and  ilities.  Its  analysis  of  the  current  state  of  the  SQs  art  and  practice  has  identified  some 
serious  shortfalls  in  their  structure  and  coverage,  which  translate  into  serious  sources  of  DoD  program 
risk. 

This  program  pursues  the  Grand  Challenge  of  performing  ilities  tradespace  and  affordability  analysis  for 
cyber-physical-human  systems  and  SoSs  in  a  portfolio  and  enterprise  context.  It  integrates  current 
strengths  in  physical-systems  tradespace  analysis  and  information-systems  tradespace  analysis.  It 
pursues  both  basic  research  on  the  foundational  relationships  among  ilities,  and  applied  research  on  the 
pilot  application  and  evolution  of  ilities  tradespace  and  affordability  MPTs  within  key  DoD  application 
domains,  including  full-coverage  cyber-physical-human  total  ownership  cost  estimation  models 
addressing  the  new  characteristics  of  future  DoD  systems,  systems  of  systems,  portfolios,  and  enterprises. 

This  research  program  primarily  implements  SEMT  Strategy  1  above,  Make  Smart  Trades  Quickly.  Table 
5.3-1  offers  a  description  of  the  Systems  Qualities  Project  (SQ),  the  one  current  Core-funded  project  in  the 
Affordability  and  Value  in  Systems  Program. 

Table  5.3-1.  Projects  in  the  Affordability  and  Value  in  Systems  Program 


Project 

Started 

Purpose 

Primary  SEMT 
Supported  Strategies 

Systems 

Qualities 

2012 

Pursue  the  Grand  Challenge  of  performing  ilities  tradespace  and 
affordability  analysis  for  cyber-physical-human  systems 

1,4 

5. 3. 3.1  System  Qualities  (SQ)  Project 

Table  5.3-2  shows  the  focus  and  deliverables  in  the  SQ  Project  through  2018.  The  project  has  three  primary 
components.  The  Foundations  component  has  pursued  three  complementary  SQ  representation 
approaches.  An  Ontology  approach  uses  DoD  stakeholder  value  propositions  to  organize  means-ends 
relationships  involved  in  satisfying  the  stakeholders'  value  propositions,  and  identifies  sources  of 
variability  in  the  SQ  values.  A  Semantic  approach  identifies  change-oriented  SQs  in  terms  of  the  semantics 
of  their  causes,  contexts,  agents,  and  effects.  A  Formal-Methods  approach  uses  precisely  defined  terms 
to  represent  the  SQs  and  their  relationships.  These  perspectives  have  been  found  to  be  complementary, 
and  efforts  are  proceeding  to  organize  them  into  a  unified  framework,  and  to  use  the  framework  to 
organize  guidance  for  systems  engineers  in  balancing  the  tradeoffs  among  the  SQs. 

The  current  initial  form  of  the  stakeholder  value-based,  means-ends  framework  has  Stakeholder 
Satisfaction  as  its  ultimate  objective,  and  the  systems  engineering  of  successful  cyber-physical-systems  as 
its  domain.  It  includes  the  stakeholder  values  of  having  current-system  Mission  Effectiveness  (with 
balanced  means  of  Speed,  Delivery  Capability,  Accuracy,  Usability,  Scalability,  and  Versatility);  current- 
system  Efficiency  (with  balanced  means  of  Cost,  Duration,  Personnel,  and  other  Scarce  Quantities, 
Producibility,  and  Supportability);  current-system  Dependability  (with  balanced  means  of  Reliability, 
Availability,  Maintainability,  Safety,  Security,  Privacy,  Robustness,  and  Survivability);  along  with  future- 
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system  Flexibility  (with  balanced  means  of  Modifiability,  Adaptability,  and  Composability).  Further  SQs 
such  as  Extendibility,  Understandability  and  Testability  are  lower-level  means  supporting  one  or  many  of 
the  means  above. 


Table  5.3-2.  System  Qualities  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Research  and  develop  basic  system  qualities  (SQs) 
concepts  and  framework.  Explore  early  MPT 
applications  and  interoperability,  including  with 

ERS  counterparts. 

Basic  SQ  concepts  and  framework.  Results  of 
early  MPT  applications  and  Valuing  Flexibility 
approaches.  Initial  definition  of  integrated 
lifecycle  cyber-physical-human  system  cost 
model. 

2014 

Explore  integration  of  multi-view  SQ  ontology 
definitions.  Elaborate  SQ  concepts  and  framework 
in  key  areas;  e.g.,  SoSs,  set-based  design. 

Hold  community  workshops  on  next -generation 
cost  model  elements. 

Multi-view  SQ  ontology  definition  papers. 

Results  of  broader  and  deeper  MPT 
applications.  Initial  need/data  prioritization  of 
next-gen  cost  models  for  software,  SE.  Early 
software  cost  estimation,  metrics  manual  v0.5. 

2015 

Integrate  SQ  ontology  views  into  unified  SQ 
Ontology;  prototype  initial  support  tools. 

Increasingly  scalable  and  interoperable  SQ  MPTs, 
set-based  design  aids.  Next-generation  SE  and 
software  cost  models  defined;  initial  data 
collection. 

Mature  SQ  Ontology  Framework.  Results  of 
increasingly  scalable  and  interoperable  MPT 
applications.  Exploratory  application  of  set- 
based  design  concepts  with  the  Army  Tank  and 
Automotive  Research,  Development,  and 
Engineering  Center  (TARDEC)  and  the  Naval  Sea 
Systems  Command  (NAVSEA).  Publication  of 
Early  Software  Cost  Estimation  and  Metrics 
Manual. 

2016 

Experimentally  apply,  refine  unified  SQ  Ontology, 
and  associated  support  tools.  Elaborate  Synergies 
and  Conflicts  matrix.  Initial  research  in  automated 
diagnosis  of  SQ  levels.  Experimentally  apply,  refine 
scalable,  interoperable  MPTs,  set-based  design 
aids.  Explore  domain-specific  extensions, 
collaborations  with  other  initiatives.  Initial 
calibration  of  Next-Generation  Cost  Models;  use  in 
extensions  to  full  life-cycle  models. 

SQ  Analysis  Guidebook  v  1.0,  associates 
support  tools.  Beta-test  versions  of  SQ 

Synergies  and  Conflicts  analysis  tools.  Scalable 
and  interoperable  SQ  MPTs.  Initial  calibration 
of  Next-Generation  Cost  Models.  Initial 
exploratory  full  life  cycle  cost  estimation 
model. 
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Year 

Focus 

Key  Deliverables 

2017 

Experimentally  apply  and  refine  SQ  Ontologies  and 
Synergies  and  Conflicts  support  tools.  Prototype 
wiki-based  Qualipedia.  Use  as  basis  for  research 
and  development  automated  Synergies  and 

Conflicts  analysis  and  diagnostic  tools.  Increasingly 
scalable  and  interoperable  MPTs.  Further  set- 
based  design  results  and  applications.  Guidebook- 
based  outreach  and  educational  initiatives. 

New-idea-extended  SQ  concepts  and 
framework.  Results  of  increasingly  scalable  and 
interoperable  MPT  applications.  Extended 
multi-domain  lifecycle  cyber-physical-human 
system  cost  model,  SQ  Analysis  Guidebook  v 

1.5.  Initial  Guidebook-oriented  courseware. 

2018 

Evaluate,  extend,  and  refine  Qualipedia,  initial 
automated  S&C  analysis  and  diagnostic  tools. 
Converge  unified  approach  to  set-based  design. 
Extend  results  to  portfolio  and  enterprise  levels. 
Productize  Qualipedia-based  educational 
technology,  disseminate  via  short  courses  for  DoD, 
faculty,  industry. 

Initial  production-grade  Qualipedia,  including 
initial  portfolio  and  enterprise  extensions. 
Scalable  automated  Synergies  and  Conflicts 
analysis  and  diagnostic  tools. 

Qualipedia-based  courseware,  initial  usage  at 

Air  Force  Institute  of  Technology,  Naval 
Postgraduate  School,  DAU,  SERC,  and  other 
universities. 

The  second  primary  component  involves  extending  and  integrating  existing  SQ  MPTs  to  better  support 
DoD  cyber-physical-human  SQanalysis.  This  includes  developing  more  service-oriented  and  interoperable 
versions  of  current  SERC  SQ  MPTs;  developing  approaches  for  better  integrating  MPTs  primarily  focused 
on  physical,  cyber,  or  human  system  SQ  analysis;  efforts  to  modify  and  compose  existing  SERC  SQ  MPTs 
to  better  interoperate  with  each  other  and  with  counterpart  MPTs  in  the  ERS  community  and  elsewhere; 
and  efforts  to  apply  the  MPTs  to  the  ilities  tradespace  and  affordability  analysis  of  increasingly  challenging 
DoD  systems. 

The  third  primary  component  focuses  on  affordability  analysis.  It  addresses  the  challenges  of  cost 
estimation  for  the  next  generation  of  DoD  systems  in  support  of  the  DoD  emphasis  on  affordability  in  the 
BBP  memoranda.  These  challenges  include  costing  of  more  incremental  and  evolutionary  development 
approaches,  of  increasingly  interdependent  systems  of  systems,  of  agile  development  of  rapidly  fielded 
systems,  of  increasingly  autonomous  systems,  and  of  the  tradespace  among  development  costs, 
operations  and  sustainment  costs,  and  the  costs  of  achieving  higher  or  lower  levels  of  system  qualities. 

The  SQ  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.3-3  and  5.3-4  on  the  next 
page. 
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Table  5.3-3.  SQ  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Engage  collaborative  organizations  in  DoD  (Army  Engineer  Research  and 
Development  Center,  TARDEC;  Naval  Air  Systems  Command,  NAVSEA,  Marine 
Corps,  Air  Force  Aeronautical  Systems  Center  and  Space  and  Missile  Systems 
Command);  Industry  (major  aerospace  companies,  cost  model  proprietors), 
FFRDCs  (Aerospace,  Software  Engineering  Institute);  Professional/Industry 
Societies  (INCOSE,  International  Software  Engineering  Research  Network,  NDIA) 
in  exploring  and  prioritizing  technical  approaches 

•  Engage  Community 

2 

Organize  project  into  top-priority  focus  areas:  SQ  Ontology  and  Guidance;  SQ 
Models  and  MPTs;  Cost  and  Schedule  Modeling 

•  Plan  Early 

•  Engage  Community 

•  Balance  Long  and 

Short  Term 

3 

Develop  interoperable,  service-oriented  models  and  MPTs:  converged  general, 
change-oriented,  and  formal  ontologies;  interoperable  set-based  design  aids, 
SysML  models  and  cost  estimating  tools;  Air  Force  and  Navy  intelligence, 
surveillance  and  reconnaissance  UAV  models  and  MPTs;  FACT-based  multi- 
Service  models  and  MPTs  and  Cost-Schedule  estimate-range  models  and  MPTs, 
emphasizing  uncertainties  and  risks. 

•  Productize 

•  Support  Centrally 

4 

Use  workshops,  prototypes,  and  pilots  to  engage  stakeholder  communities  in 
exploring,  evaluating,  and  evolving  increasingly  relevant  and  practical  models 
and  MPTs 

•  Engage  Community 

•  Pilot  Continuously 

Table  5.3-4.  SQ  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Results  of  spearheading  SQ  Community  of  Interest  in  identifying,  engaging,  and 

collaborating  with  SQ  stakeholders  in  exploring,  evaluating,  and  evolving  increasingly 
relevant  and  practical  models  and  MPTs,  using  evidence-based  4-step  plan  above. 

Progress 

(approval, 

adoption) 

•  9  papers  published  in  the  Proceedings  of  CSER  2014  and  2015,  and  in  INCOSE  International 
Symposia  2014  and  2015. 

•  8  presentations  at  NDIA  SE  Symposia  2014,  2015.  Several  presentations  at  NDIA-Army 
Ground  Vehicle  SE  and  Technology  Symposia  2014,  2015.  SERC  Best  Student  Paper  2014. 

•  Workshops  with  industry  and  Government  at  Aerospace  Corp. 

•  Ground  Systems  Architecture  Workshops  2014,  2015, 

•  Army  Practical  Systems  and  Software  Measurement  Workshop  2015, 

•  Systems  and  Software  Cost  Modeling  Forums  at  the  University  of  Southern  California  2014, 

CMU-SEI  2015. 

5.3.4  Quantitative  Risk  Program 

The  Quantitative  Risk  Program,  which  currently  has  one  project,  primarily  implements  SEMT  Strategies  1 
and  3,  Make  Smart  Trades  Quickly  and  Balance  Agility  and  Assurance,  respectively. 

Table  5.3-5  describes  the  Quantitative  Technical  Risk  (QTR)  Project. 
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Table  5.3-5.  Projects  in  the  Quantitative  Risk  Program 


Projects 

Started 

Purpose 

Primary  SEMT 
Supported  Strategies 

Quantitative 

Technical 

Risk  (QTR) 

2012 

Develop  a  mix  of  near-term  and  long-term  MPTs  that 
quantify  the  technical  risk  programs  face  to  support 
improved  decision  making  on  how  to  address  those  risks 
early  in  the  program  lifecycle 

1,3 

5. 3.4.1  Quantitative  Technical  Risk  (QTR)  Project 

Table  5.3-6  shows  the  focus  and  deliverables  in  the  QTR  Project  through  2016.  The  timeline  beyond  2016 
has  not  yet  been  established,  and  will  be  structured  to  build  on  progress  through  2016.  Technical  risk 
refers  to  risks  that  originate  in,  have  effects  on,  or  involve  in  risk  mitigation  during  system  development 
including  system  configuration,  architecture,  baseline,  technologies,  design,  manufacturing  and/or 
integration.  The  QTR  Project  has  two  primary  foci,  to  be  addressed  in  parallel. 

The  first  focus  is  incremental  development  of  QTR  MPTs  addressing  needs  and  gaps  within  the  DoD 
acquisition  Risk  Management  process,  starting  with  "low-hanging  fruit"  then  progressing  to  more 
challenging  issues.  The  approach  is  to  involve  end-users  in  the  services  as  co-developers  and  transition 
partners  in  order  to  ensure  that  the  QTR  are  relevant  to  program  risk  issues;  practical  within  the 
acquisition  decision  and  information  structures,  processes,  and  temporal  sequences;  and  can  be 
effectively  transitioned  to  end-users.  The  intent  is  to  employ  evidence-based  methods  to  identify  high- 
impact  and  high-contribution  MPTs. 

The  second  focus  is  fundamental  research  needed  to  develop  QTR  MPTs  addressing  challenging  DoD  Risk 
Management  needs  and  issues.  The  approach  is  to  adapt  and  expand  promising  theoretical  frameworks 
and  MPT  approaches  from  areas  outside  of  traditional  DoD  risk  management,  such  as  insurance 
underwriting  for  large-scale  engineering  and  construction  projects,  real  options  in  product  design  and 
development,  and  predictive  analytics  in  insurance  and  business  development.  The  scope  includes  risks 
over  the  entire  system  life  cycle,  peculiarities  by  type  of  system,  type  of  acquisition,  type  of  cause,  type 
of  causal  chain,  and  types  of  mitigation  strategies. 
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Table  5.3-6.  Quantitative  Technical  Risk  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Framing  the  issues,  opportunities  and 
technical  approaches  for  relevant, 
practical  QTR  MPT  near-  and  long-term 
research,  development,  validation  and 
transition 

Research  framework  of  near-  and  long-term  risk 
issues  &  potential  QTR  MPT  for  DoD  acquisition. 
Findings  from  investigation  of  complexity  theory  for 
acquisition  program  risk  management.  List  of 
approaches  from  outside  traditional  DoD 
acquisition.  Plan  for  incremental  development, 
validation  and  transition.  Identification  of  potential 
case  study  systems  and  data  sources.  Outlines  of 
strategies  toward  developing  an  adaptable 
integrated  risk  management  framework,  and 
transitioning  MPT  to  end-user  systems. 

2014 

Incremental  development,  validation  and 
transition  of  QTR  MPT  in  parallel  with 
longer-term  research  for  additional  QTR 
MPT. 

Data  for  one  or  more  historical  systems  and 
acquisition  programs  for  use  in  case  studies. 
Documentation  of  one  or  more  initial  QTR  MPT 
harvesting  "low-hanging-fruit",  with  case  studies 
applying  the  initial  QTR  MPT  to  a  historical 
acquisition  program.  Technical  interface 
specifications  and  coordination  milestones  toward 
transition  initial  QTR  MPT  to  a  service  SE  and  risk 
management  system.  Identification  and 
assessment  of  "flexibility  and  adaptability" 
strategies  and  risk-hedges  (real  option)  in  DoD 
acquisition.  Identification  and  assessment  of  Risk 
Breakdown  Structures,  Risk  Factor  Analysis  and  Risk 
Estimating  Relationships  (for  predictive  analytics) 
for  DoD  programs. 

2015  -  2018 

Pilot  application,  verification  and 
validation.  Methods  to  improve  visibility 
of  risks  through  CDRL  data  reporting 
Continuing  refinement  of  QTR  MPT  in 
parallel  with  longer-term  research  on 

QTR  MPT  on  risks  in  specifications  for 
software  intensive  and  complex  cyber¬ 
physical  systems. 

Documentation  of  provisional  QTR  MPT  and 
justification/derivation.  Assessment  and 
documentation  of  pilot  application  to  a  Major 

Defense  Acquisition  Program  from  Preliminary 

Design  Review  through  Functional  Configuration 
Audit.  Community  engagement  workshops. 
Recommendations  for  Contract  Data  Requirements 
List  content  and  schedule  language  to  improve 
visibility  into  risk  exposure.  QTR  MPT  for  risk 
leading  indicators  and  underlying  theory  addressing 
errors  of  omission  and  commission  in  specifications 
for  software  intensive  and  complex  cyber-physical 
systems. 

The  QTR  Project  transition  plan  and  characterization  are  shown  in  Tables  5.3-7  and  5.3-8  on  the  next  page. 
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Table  5.3-7.  QTR  Project  T ransition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Engage  members  of  the  risk  management  community  in  discussions  and 
workshops  to  identify  high-priority  gaps  in  risk  management  capabilities, 
methods,  procedures,  and  tools,  and  to  examine  progress 

•  Engage  Community 

•  Pilot  Continuously 

2 

Engage  collaborating  agencies,  specifically  the  TARDEC  Risk  Management 

Group,  to  refine,  develop,  and  verify  suitability  of  Risk  Leading  Indicators  for 
major  acquisition  programs  and  technology  demonstrator  programs 

•  Engage  Community 

•  Pilot  Continuously 

3 

Discriminate  between  near-term  opportunities  for  practical  Risk  Leading 
Indicators  from  challenging  risk  sources  for  longer-term  research  (e.g.,  risk 
metrics  for  errors  of  omission  and  commission  the  early  specifications  of 
software-intensive  and  cyber-physical  systems) 

•  Long  and  Short 

Term 

4 

Conduct  a  pilot  application  of  Risk  Leading  Indicators  with  a  current  Army 
major  acquisition  program  to  test  and  demonstrate  practicality  with 
information  available  during  development,  value  and  relevance  to  the  program 

•  Pilot  Continuously 

•  Engage  Community 

5 

Formulate  recommendations  for  Contract  Data  Reporting  schedule  and 
content  that  would  improve  visibility  into  program  risk  exposure 

•  Productize 

Table  5.3-8.  QTR  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Active  engagement  with  the  community  in  developing,  piloting,  and  assessing  the 

research  products  with  evidence  based  assessment 

Progress 

•  Extensive  collaboration  with  TARDEC  on  program  risk  assessment. 

(approval, 

•  Several  presentations  at  NDIA-Army  Ground  Vehicle  SE  and  Technology  Symposia  2014, 

adoption) 

2015. 

5.3.5  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  Program 

The  IMCSE  Program  will  include  and  significantly  extend  the  traditional  focus  on  the  modeling  of  system 
products  and  the  use  of  the  models  to  perform  system  qualities  (SQ)  tradespace  and  affordability  analyses 
as  described  above,  and  increasingly  use  models  to  generate  software  and  hardware  products.  The  IMCSE 
Program  extensions  will  address  the  modeling  of  system  execution  processes,  such  as  operational  concept 
formulation,  and  system  development  processes,  which  can  also  be  executed  to  aid  in  the  generation  of 
system  products.  Further,  as  was  emphasized  in  the  IMCSE  section  of  the  SERC  Systems  2020  Report,  an 
additional  focus  on  modeling  the  system's  environment  will  be  pursued,  which  is  needed  for  performing 
many  of  the  SQ  tradespace  and  affordability  analyses.  Models  can  also  improve  affordability  by 
automatically  generating  needed  documentation,  or  even  better  by  serving  as  the  documentation  itself. 
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Further,  models  can  reduce  or  avoid  system  overruns  and  performance  shortfalls  by  enabling  more 
thorough  Analyses  of  Alternatives  and  evidence-based  decision  reviews. 

This  research  program,  which  currently  has  one  Core-funded  project,  primarily  implements  all  four  of  the 
SEMT  strategies.  It  builds  on  an  earlier  project,  Graphical  Concept  of  Operations,  and  on  the  IMCSE  project 
currently  underway  that  began  in  late  2013.  Table  5.3-9  summarizes  the  active  project  and  the  strategies 
it  primarily  supports. 


Table  5.3-9.  Projects  in  the  IMCSE  Program 


Projects 

Started 

Purpose 

Primary  SEMT 
Supported  Strategies 

Interactive  Model- 
Centric  Systems 
Engineering 

Late  2013 

Use  models  to  drive  systems  engineering, 
development,  production,  and  evolution 

1,  2,  3,  4 

5. 3. 5.1  Interactive  Model-Centric  Systems  Engineering  Project 

Models  have  significantly  changed  SE  practice  over  the  past  decade.  Most  notably,  model-based  systems 
engineering  (MBSE)  methods  and  tools  are  increasingly  used  throughout  the  entire  system  lifecycle  to 
generate  systems,  software  and  hardware  products,  and  replacing  labor-intensive  and  error-prone 
documentation-based  processes  with  model-based  ones.  While  substantial  benefits  have  been  achieved, 
the  most  impactful  application  of  models  in  SE  has  yet  to  be  realized.  Truly  transformative  results  will 
only  come  through  intense  human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with 
models  in  order  to  make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and 
future  uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints.  The  IMCSE  Project  seeks 
to  enable  this  transformation. 

The  complexity  and  socio-technical  nature  of  contemporary  systems/SoSs  drives  an  urgent  need  for  a 
more  powerful  integration  of  humans  and  technologies.  Early  concept  decisions  have  always  been 
critically  important,  and  with  continuously  evolving  SoSs  having  long  life  spans,  such  decisions  are  now 
made  throughout  the  entire  life  cycle.  Soft  factors  become  increasingly  influential.  For  example,  trust  in 
model-based  data  sets  and  decisions  are  in  part  determined  by  the  chosen  model  itself  as  perceived  by 
specific  decision  makers.  The  timescale  of  making  early  architectural  decisions  is  out  of  sync  with  the 
current  model-based  systems  engineering  capabilities  and  decision  environments.  New  algorithms  and 
novel  modeling  approaches  must  be  discovered  to  accelerate  technical  and  programmatic  decision 
support  from  months  to  minutes;  in  order  to  effectively  leverage  and  incorporate  human  knowledge  and 
judgment,  this  requires  an  interactive  capability.  Much  potential  exists  in  maturing  emerging  novel 
methods  for  evaluating  system  responsiveness  under  complex  uncertainties,  to  enable  engineering  of 
resilient  systems.  Further,  as  was  emphasized  in  the  SERC  Systems  2020  Report,  modeling  the  system's 
dynamic  operational  environment  remains  an  open  area  of  research.  Forward  research  is  informed  by  a 
recently  completed  SERC  Graphical  Concept  of  Operations  Project. 

The  IMCSE  Project  involves  three  tasks  initiated  in  2014,  followed  by  additional  tasks  in  out-years.  The 
three  tasks  are: 
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1.  Pathfinder.  The  task  will  investigate  the  current  state  of  the  art/practice  in  interactive  model¬ 
centric  systems  engineering.  Surveys  and  literature  review  were  used  to  establish  a  preliminary 
picture  of  what  is  being  done  in  practice,  current  MPTs,  and  what  research  has/is  being 
performed.  This  informed  the  planning  and  conduct  of  an  invited  workshop  to  identify  research 
opportunities,  gaps  and  issues. 

2.  Interactive  Schedule  Reduction  Model.  This  task  builds  on  an  existing  prototype  model  (prior 
Defense  Advanced  Research  Projects  Agency  support)  for  interactively  exploring  alternatives  in 
the  systems  development  process  and  application  of  resources.  The  model  enables  rapid 
sensitivity  analysis  of  various  factors  to  determine  their  potential  impact  on  program  schedule. 
Exploratory  extensions  of  the  model  will  be  developed  and  evaluated,  resulting  in  a  new 
prototype  for  pilot  application. 

3.  Interactive  Epoch-Era  Analysis.  This  task  involves  research  to  extend  a  current  approach  for 
evaluating  systems  under  uncertainty,  Epoch-Era  Analysis,  through  the  development  of  an 
interactive  capability.  The  resulting  prototype  method  and  supporting  tools  are  being  applied  to 
a  case  on  uncertainties  in  mission  planning  and  deployment  support,  of  particular  interest  to  the 
ERS  program.  This  case  application  serves  as  a  pathfinder  for  identifying  key  considerations  for 
applicability  and  deployability  of  the  method  for  eventual  DoD  use. 


Table  5.3-10  shows  the  focus  and  deliverables  in  the  IMCSE  Project  through  2018  aimed  at  addressing  the 
three  tasks  above,  as  well  as  new  tasks  in  out-years.  In  2014,  the  "Pathfinder"  task  investigation  identified 
synergistic  research  opportunities.  Beginning  with  the  initial  2014  workshop,  periodic  targeted  workshops 
will  be  convened  with  the  intent  of  examining  ongoing  research  that  can  be  leveraged  in  the  SERC  research 
efforts,  and  involving  the  broader  community  in  creating  -  and  realizing  -  the  vision  and  research  agenda 
for  the  IMCSE  project  and  the  broader  program. 
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Table  5.3-10.  IMCSE  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

New  start 

2014 

Pathfinder  task  with  collaborative  research 
discovery;  exploratory  extensions  to  an 
existing  development  schedule  reduction 
model;  exploratory  piloting  and  interactive 
extensions  to  Epoch-Era  method 

Pathfinder  Task:  Workshop  to  explore  issues  and 
opportunities,  with  report  on  workshop  results. 
Pathfinder  task  report,  with  findings  of  research 
opportunities,  gaps,  and  issues.  Out-year  research 
plans  based  on  pathfinder  results. 

Interactive  Schedule  Reduction  Model:  Exploratory 
extensions  implemented  and  evaluated.  Report  on 
exploratory  schedule  model.  Prototype  model  for 
pilot  application. 

Interactive  Epoch-Era  Analysis:  Exploratory  research 
to  develop  interactive  capability,  with  demonstration 
via  a  mission  planning  support  application  case. 

Report  on  exploratory  research  and  case  application. 

2015 

Initiate  multi-year  research  plans  based  on 
Pathfinder  results,  including  2014  follow-on 
for  one  or  both  of  the  exploratory  research 
tasks.  Assess  results  individually  and 
comparatively. 

IMCSE  Project  Applications:  Based  on  Pathfinder 
results,  select  and  initiate  one  or  more  additional 
tasks,  and  increase  SERC  member  collaboration  in 
tasks.  Report  will  document  the  maturation  of  the 

MPTs  for  each  of  these  projects,  with  comparative 
results. 

2016 

Increasing  maturation  of  IMCSE  MPTs  and 
enabling  environments,  leading  to  adoption 
by  user  community;  extend  IMCSE  scope 
via  increased  collaboration  of  universities 
and  broader  user  community.  Exploration 
of  further  new-idea  projects. 

IMCSE  MPT  Implementations  and  Impact 

Assessments:  Continued  maturation  and 
implementation  of  IMCSE  MPTs,  with  enabling 
environments.  Ongoing  study  of  impacts  resulting  in  a 
comprehensive  report  of  progress,  results,  and 
opportunities. 

2017  -  2018 

Increasing  maturation  and  synthesis  of 

IMCSE  MPTs  and  enabling  environments, 
leading  to  adoption  by  user  community; 
sustain  and  increase  collaboration  of 
universities  and  broader  user  community. 
Step-ups  of  new-idea  tasks. 

IMCSE  MPT  Synthesis  Impact  and  Effective  Practice 
Assessments:  Continued  maturation,  synthesis  and 
implementation  of  IMCSE  MPTs,  with  enabling 
environments.  Ongoing  study  of  real-world  impacts  to 
identify  successful  practices.  A  comprehensive  report 
of  impacts  and  insights,  with  guidance  on  practice. 
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The  IMCSE  transition  action  plan  and  characterization  are  shown  in  Tables  5.3-11  and  5.3-12  below. 


Table  5.3-11  IMCSE  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

The  IMCSE  Project  has  been  developed  using  three  complimentary  thrusts 
(foundations,  fundamentals,  applications)  with  different  timescales,  to  have 
impact  on  the  long  term,  near  term  and  the  present.  An  over-arching  project 
goal  is  to  build  a  community  of  interest  around  IMCSE. 

•  Plan  Early 

•  Balance  Long  and 
Short  Term 

•  Engage  Community 

2 

An  IMCSE  Pathfinder  Workshop  engaged  members  of  the  community  in 
characterizing  state  of  the  art/practice,  identifying  research  needs,  and 
envisioning  the  model-centric  environment  of  the  future.  A  workshop  report 
was  published  and  distributed.  Efforts  to  gather  research  needs  and  develop 
broader  collaboration  on  longer  term  research  agenda  have  been  initiated. 

•  Engage  Community 

•  Balance  Long  and 
Short  Term 

3 

A  proof-of-concept  prototype  for  the  Interactive  Schedule  Reduction  Model  was 
completed,  demonstrated  to  practitioners,  and  software  made  available 
through  a  website. 

•  Engage  Community 

•  Pilot  Continuously 

•  Productize 

4 

Prototype  visualization  tools  for  Interactive  Epoch-Era  Analysis  have  been 
piloted  with  students  and  demonstrated  to  practitioners.  An  interactive 
demonstration  prototype  is  being  made  available  online  to  gain  feedback,  with 
continuing  updates  planned  as  the  MPT  matures. 

•  Engage  Community 

•  Pilot  Continuously 

•  Productize 

5 

Model  trade-off  case  studies  for  value  models  have  been  completed  and  made 
available.  Performance  model  and  cost  model  trade-off  cases  will  be  developed 
and  made  available. 

•  Productize 

•  Pilot  Continuously 

6 

IMCSE  has  held  knowledge  exchanges,  collaborated  with,  and  discussed  future 
pilot  opportunities  with  several  other  universities,  FFRDCs  and  non-profits,  and 
government  agencies. 

•  Productize 

•  Engage  Community 

Table  5.3-12.  IMCSE  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  IMCSE  Technical  Reports  and  the  IMCSE  Pathfinder  Workshop  Report  are  available  to  the 
SERC  community.  Once  completed,  the  demonstration  prototype  with  documentation  can 
be  accessed  freely  and  downloaded.  A  second  demonstration  prototype  is  in 
implementation. 

Progress 

(approval, 

adoption) 

•  IMCSE  has  effort  has  resulted  in  2  technical  reports,  3  published  conference  papers,  5 
conference  presentations,  and  held  one  workshop  (with  report  published). 

•  An  IMCSE  paper  received  the  2014  SERC  Best  Student  Paper  Award  and  another  IMCSE 
paper  received  the  CSER  2015  Best  Academic  Paper  Award 

5.3.6  Agile  Systems  Engineering  Program 

Agile,  lean,  and  other  adaptive  processes  and  governance  mechanisms  are  key  to  meeting  the  SEMT  Grand 
Challenge.  To  be  adaptive,  systems  engineers  must  have  MPTs  to  identify,  understand,  and  quickly  react 
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to  issues  raised  by  increasing  reliance  on  systems  of  systems,  interoperability  between  legacy  and  new 
capabilities,  evolving  requirements  throughout  the  development  lifecycle,  and  the  changing  economic 
and  political  factors  that  undergird  and  enable  system  development.  At  the  same  time,  the  human 
relationships  that  anchor  good  decision-making  depend  on  enhanced  visibility  into  increasingly  deep 
supply  chains,  constraining  contractual  relationships,  and  murky  component  systems.  Identifying  and 
validating  management  and  governance  strategies  and  mechanisms  that  inherently  support  negotiation, 
collaboration,  and  shared  outcomes  are  a  major  goal  of  this  research. 

Adaptive  processes  can  reconcile  the  differences  between  systems  engineering  and  software  engineering 
activities,  and  integrate  them  to  achieve  more  reliable  flow  and  faster  delivery  of  value  to  the  success 
critical  stakeholders.  Software  enables  rapid  deployment  of  enhanced  or  new  capabilities  while 
maintaining  continuous  overall  system  operations.  Currently,  software  development  processes  do  not 
operate  seamlessly  with  SE  processes.  As  an  example,  the  ability  for  software  to  provide  incremental 
capability  requires  adaptation  of  SE  MPTs  to  enable  modularity,  flexibility,  continuous  integration,  and 
incremental  verification  and  validation.  The  Agile  SE  Program  seeks  balance  between  the  integrity  and 
stability  of  traditional  SE  and  the  rapid  response  and  immense  flexibility  of  software  engineering.  Having 
gained  significantly  better  understanding  of  the  issues  and  concerns  facing  SE  management  and 
government,  the  two  projects,  Kanban  in  SE,  and  Agile  SE  Enablers  and  Quantification  work  closely 
together. 

With  a  mix  of  Core  and  non-Core  funding  in  2015  to  support  the  development  of  the  Demonstration  and 
Analysis  Tool  for  Adaptive  Systems  Engineering  Management  (DATASEM),  the  two  current  projects  have 
worked  closely  in  the  definition  and  development  of  the  DATASEM  environment.  The  Agile  SE  Enablers 
and  Quantification  Project  undergirds  and  drives  a  broader  scope  of  adaptive  SE  and  SoSE  research 
activities.  Both  will  share  participation  in  and  leverage  INCOSE's  Agile  SE  Life  Cycle  Model  Fundamentals 
project14.  As  illustrated  in  Figure  5.3-1,  the  identification  of  SE  enablers  will  be  a  coordinated  partner  with 
the  Kanban  in  SE's  DATASEM  evolution  and  transition  process. 


14  See  http://www.incose.org/newsevents/news/2014/ll/17/domain-independent-agile-se-life-cycle-proiect. 
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Figure  5.3-1.  Agile  SE  Management  Project  Updates 


Table  5.3-13  offers  a  description  of  the  projects— completed,  completing,  and  new— and  the  strategies 
they  primarily  support. 
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Table  5.3-13.  Projects  in  the  Agile  SE  Program 


Project 

Started 

Purpose 

Primary 

SEMT 

Supported 

Strategies 

Kanban  in  SE 

2010 

Exploration  of  applicability  of  Kanban  methods  to  complex  system 
evolution  where  development  is  primarily  through  capability 
enhancement  and  new  features  added  to  an  existing  system. 
Providing  tools  that  support  experimentation,  validation,  and 
transition  of  defense  programs  to  effective  adaptive  SE 
management  and  governance  approaches.  Development  and 
evolution  of  DATASEM,  and  extending  the  DATASEM  scope  to 
evolving  operational  environments. 

2,3,4 

Agile  SE 
Enablers  and 

Quantification 

2008  and 

continued 

in  2012 

Identifying  and  employing  research  that  can  inform  and  improve 
agile  (adaptive)  SE,  drawing  on  the  SERC  RT-34  Expediting  SE  effort 
and  its  initial  model  quantifying  agile  schedule  compression. 

Providing  a  growing  set  of  management  and  governance 
mechanisms,  technical  processes,  work  flows  and  simulation 
capabilities  that  reflect  new  adaptive  systems  engineering  enablers 
and  practices. 

1,2, 3,4 

5.3.6. 1  Kanban  in  SE  Project 

Initially  motivated  by  the  ineffectiveness  of  integrated  master  schedules  in  rapidly  changing  operational 
environments  and  the  success  of  Kanban  approaches  for  the  knowledge  work  of  software  development, 
the  project  has  expanded  to  investigate  the  applicability  of  Kanban  as  well  as  that  of  other  adaptive 
governance  mechanisms.  The  initial  Kanban  scheduling  system  (KSS)  networks  described  in  SERC  RT-35 
sought  a  way  to  prioritize  engineering  tasks  based  on  SoS  or  complex  system  capability  priorities  and  task 
interdependencies  by  selecting  value-adding  features  first,  reducing  wait  time  for  scarce  engineering 
specialties,  and  minimizing  time  wasted  on  context  switching  by  overloaded  resources.  The  KSS  network 
concept  provides  two  valuable  side  effects.  First,  the  implementation  of  the  network  supports  critical 
conversations  about  schedule  and  value  decisions  by  the  appropriate  people  at  the  right  time  and  nearest 
the  actual  implementation.  Second,  the  network  significantly  improves  executive  and  systems  engineering 
visibility  into  the  status  of  multiple  independent  development  organizations. 

Research  in  2014-2015  developed  DATASEM,  a  simulation  generator  and  test  bed  for  experimenting  with 
the  KSS-introduced  and  related  adaptive  governance  mechanisms.  DATASEM  uses  a  set  of  Domain  Specific 
Languages  (DSLs),  created  to  describe  organization,  work  flow,  and  governance  models,  to  generate 
simulations  for  experimentation  and  demonstration.  Initial  versions  of  DSL  have  been  completed,  and 
translation  and  implementation  code  developed  for  simulating  the  core  KSS  mechanisms  using  an  agent- 
based  simulation  facility  (Repast).  Tools  to  develop  experiments,  define  mechanisms,  and  display 
simulation  results  are  also  being  developed.  A  first  generation  DATASEM  suite  was  completed  in  2015. 

The  first  generation  DATASEM  Suite  will  provide  significant  functionality  and  information.  However,  only 
a  subset  of  possible  models  and  mechanisms  will  have  been  developed,  and  the  DSLs  and  infrastructure 
can  be  extended  to  address  a  variety  of  additional  investigations.  There  are  numerous  opportunities  to 
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extend  the  capabilities  and  enhance  the  usefulness  of  DATASEM  including  a  wider  range  of  agile  methods, 
incorporating  technical  debt,  as  a  demonstration  vehicle  for  the  impact  of  changing  governance 
mechanisms,  modeling  of  acquisition  processes;  and  identifying,  evaluating  and  implementing  new 
mechanisms  and  approaches  that  emerge  from  these  research  efforts.  For  example: 

•  DATASEM  has  attracted  significant  attention  from  the  system  of  systems  community  as  a  way  to 
not  only  investigate,  but  also  demonstrate  the  impact  of  change  on  both  existing  and  adaptive 
governance  mechanisms,  and  to  measure  the  effectiveness  of  various  combinations.  Adding  to 
the  DSL  and  infrastructure  support  for  modeling  organizational  negotiation  and  communication 
activities  related  to  SoS  governance  can  improve  management  understanding  of  the  critical 
nature  of  such  activities  to  effective  governance. 

•  DATASEM  could  be  applied  to  acquisition  governance  processes  to  develop  and  test  alternative 
governance  approaches  and  to  identify  and  create  ways  to  mitigate  incompatibilities  between 
adaptive  and  traditional  governance  mechanisms.  Industry  has  suggested  the  project  consider 
the  concept  of  technical  debt  in  the  DATASEM  models  and  as  a  significant  attribute  for 
characterizing  particular  configurations  of  workflow,  organization,  and  governance,  leveraging 
other  SERC  research  in  this  area. 

•  INCOSE  is  conducting  a  broad  analysis  of  the  agile  systems  engineering  life  cycle.  This  should 
identify  additional  mechanisms  and  approaches  that  are  not  directly  supported  by  the  first 
generation  and  provides  a  way  of  validating  some  of  the  first  generation  mechanism  simulations. 

Table  5.3-14  shows  the  focus  and  deliverables  in  the  project  through  2018. 


Table  5.3-14.  Kanban  in  SE  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Studying  on-demand  scheduling  systems  and 
adapting  the  concept  to  SE 

KSS  concepts;  RT-35  Kanban  in  SE  reports  and 
papers;  Kanban  in  SE  industry  working  group 

2014-2015 

Building  the  first  generation  DATASEM  Suite 

Fully  operational  stand  alone  and  web-available 
versions 

2016 

Building  an  empirical  database  and 
components  to  support  further 
experimentation  and  evolution;  extending 
application  contexts;  continuing  SE  enabler 
identification,  evaluation,  and  implementation 
in  DATASEM 

Refinements  of  first  generation  components; 

New  DATASEM  components  to  handle  new 
mechanisms  including  technical  debt  as  a 
comparative  indicator,  acquisition  activities; 

New  versions  of  Quantitative  Schedule 

Acceleration  Model  (QSAM)  with  expanded 
applicability;  additional  domain  coverage 

2017  -  2018 

Expansion  of  user  community  and  databases 

Expand  experimentation  and  use  for  transition  of 
industry  and  government  to  more  adaptive 
governance  as  appropriate 
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The  Kanban  in  SE  transition  plan  and  characterization  are  shown  in  Tables  5.3-15  and  5.3-16  below. 


Table  5.3-15.  Kanban  in  SE  Transition  Project  Transition  Plan 


# 

Transition  Action 

Principles  Implemented 

1 

Work  in  direct  response  to  customer  needs  with  the  goal  of  improving  SoS  SE 
performance.  Industry  and  government  volunteers  involved  in  the  definition  and 
substance  of  the  concept  developed  through  an  industry-working  group. 

•  Plan  Early 

•  Engage  Community 

2 

The  primary  deliverable  designed  to  validate  the  concepts  and  provide  a 
persistent  and  evolving  tool  to  support  transition  by  allowing  adopters  to  "try 
before  they  buy". 

•  Balance  Long  and 

Short  Term 

•  Pilot  Continuously 

3 

As  the  tool  is  evolving,  maintaining  contact  with  industry  and  government 
through  NDIA,  and  INCOSE.  Publishing  and  participating  in  conferences  and 
workshops  to  both  demonstrate  the  capabilities  and  gather  improvement  ideas. 

•  Plan  Early 

•  Engage  Community 

4 

The  toolset  and  environment  will  be  hosted  by  the  SERC  and  available  to  the 
public.  It  will  also  be  available  as  a  package  for  use  within  restricted 
environments. 

•  Support  Centrally 

•  Productize 

Table  5.3-16.  Kanban  in  SE  Project  Transition  Plan 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  The  2014-2015  activities  produce  an  executable,  deployable,  and  useful  tool. 

•  The  2016  activities  will  implement  an  empirical  research  program  validating  the  concepts, 
measuring  sensitivities,  comparing  results  to  pilots  and  establishing  whether  the  tool 
meets  either  or  both  of  the  transition  support  and  research  goals;  it  will  also  expand  the 
usefulness  and  range  of  users  by  integrating  the  tool  with  the  Experience  Accelerator. 

Progress 

(approval, 

adoption) 

•  Kanban  in  SE  has  published  8  conference  papers,  2  articles,  3  technical  reports  and  held  2 
workshops  with  industry  and  government  participants 

•  Sponsor  demonstrations  of  DATASEM  capabilities  resulted  in  both  positive  feedback  as 
well  as  additional  ideas  for  expanding  the  capabilities 

•  Industry  and  government  members  of  the  INCOSE  Agile  SE  Life  Cycle  Model  Fundamentals 
Project  have  specifically  asked  DATASEM  and  Agile  Enablers  to  be  a  part  of  that  work 

•  DATSEM  integration  with  the  Experience  Accelerator  was  suggested  independently  by  both 
the  sponsor  and  industry 

5. 3. 6.2  Agile  SE  Enablers  and  Quantification  Project 

The  Agile  SE  Enablers  and  Quantification  Project  was  originally  established  to  continue  work  on  MPTs 
identified  in  the  RT-34  and  RT-35  efforts.  It  also  established  a  method  for  evaluating  the  promise  of  new 
methods  and  techniques.  The  conducted  research  led  to  the  improvement  and  extension  of  QSAM  and 
directly  supported  the  definition  of  the  methods  and  mechanisms  implemented  in  the  first  generation 
DATASEM  suite.  It  will  continue  to  identify  new  possibilities,  but  will  concentrate  on  the  evolution  of  these 
two  central  research  tools. 
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In  2015,  the  QSAM  was  expanded  to  include  more  specific  agile  and  lean  drivers  to  better  capture  the 
activities  associated  with  their  use  in  SE.  New  datasets  were  identified  and  used  to  further  calibrate  the 
model.  Core  funding  for  this  work  has  been  suspended. 

Table  5.3-17  shows  the  focus  and  deliverables  in  Agile  SE  Enablers  through  2015. 


Table  5.3-17.  Agile  SE  Enablers  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

MPTs  in  software 

MPT  reports  and  papers;  RT-35  phase  1  work;  RT-34  work;  support  of  INCOSE 
Agile  SE  Working  Group 

2014  -  2015 

Fundamentals  of 
agility 

White  papers:  SE  as  a  service  concept.  Governance,  organizational  and 
contracting  requirements  for  implementing  agile  SE. 

Quantitative  schedule  acceleration  model  extensions,  calibration,  behavioral 
analysis. 

5.3.7  SEMT  Area  Non-Core  Funded  Projects 

During  the  time  of  the  Technical  Plan,  the  SERC  has  executed  seven  non-Core-funded  SEMT  Area  projects 
as  briefly  described  in  Table  5.3-18  below.  One  additional  project  has  been  omitted  because  of  its 
sensitivity,  and  one  project,  Kanban  for  SE,  received  funding  from  the  Intelligence  Community  to 
complement  Core  funding. 


Table  5.3-18.  SEMT  Area  Non-Core  Funded  Projects 


Project 

Sponsor 

Description 

Virtual  Collaboration 
Environment  for  Conducting 
Project  Design  and  Tests 

Marines 

Enhance  the  FACT  tool  to  integrate  with  existing  simulation  tools  in 
order  to  be  able  to  drive  additional  capabilities  into  the  simulation 
tools 

FACT  Amphibious  Combat 
Vehicle  Analysis 

Marines 

Enhance  tooling  to  conduct  tradespace  analysis  for  large  amphibious 
combat  vehicles 

FACT  Portfolio  Management 

Marines 

Enhance  the  FACT  tool  to  be  able  to  identify  system  components 
and  correlate  them,  with  their  identified  attributes,  into  a  set  of 
capability  options 

SE  Modeling  &  Simulation 
Integration  Framework 

Army 

Review  existing  and  emerging  models,  frameworks  and  standards 
relevant  to  modeling  the  SE  workflow  for  modeling  and  simulation 
integration  in  the  digital  thread;  select  a  framework  and  develop 
preliminary  guidelines  to  model  and  document  the  digital  thread; 
and  develop  a  preliminary  model  of  how  to  transform  an 
organization  so  it  can  use  the  digital  thread  approach 

ERS  Tradespace 

Army 

Develop  decision  support  methods  and  a  tradespace  toolset 
framework  architecture  in  support  of  Engineered  Resilient  Systems 
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Project 

Sponsor 

Description 

Kanban  for  SE 

1C 

A  second  sponsor  supplemented  funding  for  the  Core-funded 

Kanban  for  SE  Project  to  support  more  automation  than  would  have 
been  possible  with  just  Core  funding. 

Transforming  SE  Through 

MBSE 

Navy 

Assess  the  technical  feasibility  of  a  radical  transformation  of 
acquisition  through  a  more  holistic  model-centric  engineering 
approach. 

5.4  Human  Capital  Development  (HCD) 

Over  the  last  decade,  DoD  and  the  defense  industrial  base  have  often  cited  a  shortfall  in  the  quantity  of 
systems  engineers  and  in  the  knowledge,  skills,  and  abilities  of  those  systems  engineers15-16.  Not  only  is 
there  a  critical  shortage  of  systems  engineers,  but  the  skill  sets  and  capabilities  of  these  engineers  need 
to  rapidly  expand  to  address  the  growing  complexity  in  the  systems  they  are  attempting  to  engineer. 
Systems  Engineering  Vision  202517  presents  a  future  view  of  SE.  That  report  highlights  several  areas  that 
directly  impact  Human  Capital  Development. 

Systems  Engineering  is  not  only  for  those  with  the  title  of  "Systems  Engineer".  Systems  skills  are  essential 
for  systems  decision  makers,  technical  leaders  and  all  engineers.  All  leaders  and  those  making  decisions 
about  systems  need  to  be  systems  thinkers.  Systems  thinking  skills  need  to  be  developed  long  before 
graduate  studies  and  should  be  introduced  as  early  as  kindergarten  through  high  school.  All  engineers 
should  have  some  education  and  training  in  systems  and  SE.  While  undergraduate  curricula  are  already 
full,  these  skills  can  be  introduced  and  distilled  in  cornerstone  and  capstone  projects.  Finally,  systems 
engineers  need  to  be  well  versed  in  a  broad  set  of  socio-technical  and  leadership  skills,  serving  as  a  central, 
multi-disciplinary  focal  point  of  systems  development  with  stakeholders  of  all  types. 

The  Human  Capital  Development  (HCD)  research  area  directly  targets  the  aforementioned  shortfalls  and 
challenges. 

HCD  Area  Goal:  Ensure  a  competitive  advantage  through  the  availability  of  highly  capable 
systems  engineers  and  technical  leaders  for  the  DoD  and  the  defense  industrial  base 


15  See 

http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/Documents/Studies/Top%20SE%20lssues%202010 

%20Report%20vll%20FINAL.pdf. 

16  See  http://www.ndia.org/Advocacv/PolicyPublicationsResources/Documents/WSARA-Public-Law-lll- 
23.pdf. 

17  See  http://www.incose.org/docs/default-source/aboutse/se-vision-2025.pdf?sfvrsn=4. 
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5.4.1  HCD  Grand  Challenge  and  Current  Progress 

The  HCD  Grand  Challenge  to  achieve  the  HCD  goal  is  to: 

Discover  how  to  dramatically  accelerate  the  professional  development  of  highly  capable 
systems  engineers  and  technical  leaders  in  DoD  and  the  defense  industrial  base  and 
determine  how  to  sustainably  implement  those  discoveries 


Significant  HCD  progress  has  been  made  through  a  mix  of  Core-funded  and  non-Core  funded  projects. 
Successes  in  the  Evolving  Body  of  Knowledge  Program  include: 

1.  The  BKCASE  Project  has  made  great  strides  in  organizing  information  and  making  it  globally 
accessible  and  available.  This  project  was  successfully  completed  and  transitioned  just  as  this 
Technical  Plan  was  approved.  Since  September  2012,  there  have  been  nearly  1,000,000  visits  to 
articles  on  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  wiki18  and  many  universities  have 
adopted  all  or  part  of  the  recommendations  found  in  the  Graduate  Reference  Curriculum  on 
Systems  Engineering  (GRCSE)19.  Their  continued  use  and  evolution  will  provide  an  up-to-date 
source  for  systems  knowledge. 

2.  As  described  in  Section  5.5. 2. 2,  the  SERC  Innovation  and  Demonstration  Lab  is  beginning  to  make 
research  resources  such  as  publications,  tools  and  data  sets  available  for  education  and  the 
advancement  of  systems  research.  This  is  an  area  that  has  great  promise  as  an  enabler  for  the 
advancement  of  systems  research  and  knowledge. 

3.  The  Helix  project  is  showing  success  in  understanding  what  enables  systems  engineers  to  be 
effective,  how  systems  engineers  mature,  and  in  characterizing  the  systems  engineering 
workforce.  Several  organizations  have  begun  using  Helix  for  their  workforce  improvement  efforts. 

Finally,  as  observed  through  BKCASE,  there  is  a  great  need  for  SE  case  studies  to  provide  real  life  lessons 
on  the  "how"  of  SE.  To  date,  it  has  been  difficult  to  acquire  the  program  data  to  provide  this  research 
with  the  desired  impact.  This  is  a  gap  that  will  need  to  be  resolved. 

The  Experience  Acceleration  (EA)  Program  has  continued  to  mature  and  now  has  a  variety  of  capabilities 
that  should  support  experiences  in  numerous  domains  and  in  several  different  single  and  multi-player 
modes.  There  is  a  great  potential  for  this  technology  to  advance  the  strategic  objective  of  educating  and 
training  faster.  Limited  pilots  have  been  conducted  that  both  show  the  potential  of  the  technology  and 
have  served  to  provide  feedback  in  its  subsequent  development.  In  addition,  a  set  of  prototype  tools  have 
been  developed  that  show  the  potential  for  tailoring  existing  experiences  and  developing  new  ones. 
Critical  work  moving  forward  is  in  learning  evaluation  and  the  validation  of  the  hypothesis  that  technology 
can  be  used  to  accelerate  learning  for  systems  thinking  and  engineering.  This  can  be  facilitated  through 
the  use  of  the  EA  in  Collaborator  university  courses  and  training.  In  addition,  it  will  be  necessary  to  show 
that  experiences  can  be  efficiently  created  and  modified  by  the  non-research  community.  Finally,  a 
sustaining  open  source  community  is  needed  to  ensure  that  Experience  Acceleration  experiences  and 
technology  can  be  supported  for  widespread  deployment. 


18  See  http://www.sebokwiki.org. 

19  See  http://www.bkcase.org. 
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The  Systems  Engineering  and  Technical  Leadership  Education  Program  continues  to  make  strides 
improving  technical  leadership  and  SE  education,  primarily  with  non-Core  funds.  The  Engineering 
Capstone  Marketplace  Project  (which  is  funded  by  a  mix  of  Core  and  non-Core  funds)  is  the  evolution  of 
research  begun  in  2010,  which  showed  that  a  multidisciplinary  senior  capstone  project  could  enhance 
development  of  SE  competencies  and  increase  interest  in  SE.  The  challenge  is  in  scaling  this  approach 
nationwide,  to  have  impact  on  how  thousands  of  students  are  taught  engineering  across  the  US.  The 
Technical  Leadership  Project  also  began  in  2010  to  evaluate  the  hypothesis  that  the  technical  leadership 
capabilities  of  high  potential,  senior  DoD  systems  engineers  and  technologists  could  be  accelerated 
through  an  educational  program  in  technical  leadership.  This  initial  research  has  spawned  several  efforts 
for  DAU  and  the  Army.  The  former  research  resulted  in  the  creation  of  an  innovative  approach  to 
educating  technical  leaders  through  three  lenses:  systems,  business,  and  enterprise.  That  approach  was 
captured  in  courses  have  been  prototyped,  piloted  and  are  being  transitioned  to  DAU.  Again,  the 
challenge  is  in  expanding  the  offering  of  these  courses  to  broaden  their  impact. 

There  are  a  number  of  additional  remaining  gaps  that  will  be  necessary  to  address  the  HCD  Grand 
Challenge.  Some  of  these  include:  how  to  better  capture  the  knowledge  of  systems  engineers  who  are 
nearing  or  in  retirement,  how  to  more  closely  couple  research  results  to  their  dissemination  in  education, 
and  how  to  expand  systems  education  into  kindergarten  through  high  school. 


5.4.2  Strategy  to  Address  the  HCD  Grand  Challenge 

Successfully  executing  the  following  strategies  will  make  significant  progress  towards  addressing  the  HCD 

Grand  Challenge: 

1.  Create  and  Provide  Easy  Knowledge  Access:  Make  it  easy  for  systems  engineers  to  understand 
the  SE  discipline  and  to  access  the  information  needed  to  expertly  perform  SE  so  that  the 
workforce  can  master  the  most  important  competencies 

2.  Educate  and  Train  Faster:  Develop  innovative  approaches  and  technology  to  educate  and  train 
systems  engineers  and  systems  teams  at  all  levels,  engineers,  and  STEM  students  much  more 
rapidly,  effectively  and  efficiently  than  with  classical  means 

3.  Develop  Effective  Technical  Leaders:  Develop  innovative  approaches  to  educate  DoD  technical 
leaders  with  the  right  mix  of  technical,  business,  and  enterprise  skills 

4.  Improve  SE  and  STEM  Education:  Develop  recommendations  and  systems  curricula  for  the  next 
generation  of  systems  engineers,  engineers  and  STEM  students 

5.  Track  Progress:  Track  the  changes  in  SE  workforce  demographics  and  performance  over  time  to 
understand  how  the  workforce  is  improving  and  how  improvement  programs  are  working 

Recall  that,  for  the  SERC,  development  of  an  MPT  includes  validation  and  transition. 

Three  HCD  research  programs  directly  implement  the  strategy: 

•  Evolving  Body  of  Knowledge 

•  Experience  Acceleration 

•  Systems  Engineering  and  Technical  Leadership  Education 
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5.4.3  Evolving  Body  of  Knowledge  Program 

This  research  program  primarily  implements  HCD  strategies  1,  4  and  5  above  -  Create  and  Provide  Easy 
Knowledge  Access,  Improve  SE  and  STEM  Education,  and  Track  Progress.  It  includes  two  current  projects 
-  Helix  and  SEEK.  A  third  project  -  BKCASE- successfully  completed  at  the  end  of  2013  as  a  research  effort, 
although  the  SERC  maintains  a  role  as  one  of  three  stewards  leading  the  operation  and  maintenance  of 
BKCASE  products.  Table  5.4-1  offers  a  description  of  these  three  projects  and  which  strategies  they 
primarily  support. 


Table  5.4-1.  Projects  in  the  Evolving  Body  of  Knowledge  Program 


Projects 

Started 

Purpose 

Primary  HCD 
Supported 
Strategies 

BKCASE 

2009; 

successfully 
completed  at 

the  end  of  2013 

Create,  disseminate,  and  evolve  the  authoritative  guide  to  the  SE 
body  of  knowledge  and  recommendations  on  curricula  for  SE 
graduate  programs 

1,  3,  4,  5 

Helix 

2012 

Understand  the  characteristics  of  the  SE  workforce  and  what 
enables  them  to  effective  and  use  those  insights  to  inform 
workforce  development  of  systems  engineers 

3,  5 

SEEK 

2013;  suspended 

at  the  end  of 

2015 

Capture  insights  from  SE  experts  about  significant  program 
successes  and  failures  and  portray  them  primarily  in  case  studies 
augmented  by  multi-media  materials  for  easy  access  and  use  in 
training,  education  and  practice 

1,3,4 

5.4.3.1  BKCASE  Project 

The  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering  (BKCASE)  Project  is  (a)  identifying 
and  making  readily  accessible  the  vast  knowledge  that  systems  engineers  need  to  know  (SEBoK)  and  (b) 
providing  recommendations  to  the  SE  academic  community  on  SE  graduate  curricula  (GRCSE).  BKCASE 
began  in  2009  as  a  SERC-supported  project  led  by  Stevens  Institute  and  the  Naval  Postgraduate  School. 
Beginning  in  2013,  INCOSE  and  the  Institute  of  Electrical  and  Electronics  Engineers  Computer  Society 
(IEEE-CS)  became  co-stewards  with  the  SERC  to  guide  and  promulgate  the  SEBoK  and  GRCSE.  Both 
products  undergo  regular  updates  to  reflect  advances  in  the  field  and  feedback  from  the  user  community. 
SEBoK  articles  have  been  accessed  more  than  1,000,000  times  since  Version  1.0  was  released  in 
September  2012.  Several  universities  in  the  US,  Europe,  and  elsewhere  have  been  adopting  GRCSE 
curriculum  recommendations.  The  SEBoK  is  novel  in  its  form  of  delivery  (a  wiki),  its  governance  model 
(shared  among  3  organizations),  its  scale  (spanning  the  technical  aspects  of  the  discipline,  how  that 
technology  is  effectively  adopted  and  used,  and  the  underlying  science  on  which  the  technology  is  based), 
and  its  rate  of  change  (multiple  updates  annually). 

No  DoD  research  funds  have  been  used  to  support  BKCASE  since  the  Technical  Plan  was  published  in 
October  2013  and  there  is  no  expectation  that  any  such  funds  will  be  used  for  BKCASE  in  the  future. 

The  BKCASE  transition  action  plan  and  characterization  are  shown  in  Tables  5.4-2  and  5.4-3  below. 
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Table  5.4-2.  BKCASE  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

From  the  very  beginning  of  the  project,  the  BKCASE  team  formed  an  alliance 
with  INCOSE  and  the  IEEE  Computer  Society  to  collaborate  on  BKCASE;  that 
agreement  led  to  participation  from  INCOSE's  corporate  members  who 
contributed  authors  and  produced  buy-in  for  them  to  adopt  both  the  SEBoK  and 
GRCSE  as  strategic  assets. 

•  Plan  Early 

•  Engage  Community 

2 

BKCASE  held  a  number  of  public  events  and  the  drafts  of  the  SEBoK  and  GRCSE 
were  widely  disseminated  for  community  review. 

•  Engage  Community 

3 

The  SEBoK  and  GRCSE  were  created,  released,  and  piloted  incrementally  in 
several  major  releases  which  were  reviewed  through  comments  provided  over 
the  Internet,  in  workshops  and  other  public  forums,  and  presented  at 
conferences.  This  also  enabled  early  adopters  to  try  draft  versions  within  their 
organizations. 

•  Pilot  Continuously 

•  Balance  Long  & 

Short  Term 

•  Engage  Community 

4 

The  BKCASE  Project  moved  from  an  effort  led  by  the  SERC  with  community 
engagement  to  a  different  governance  model  where  it  became  one  of  three 
steward  organizations.  The  governance  approach  consistently  required  all 

BKCASE  products  to  be  available  at  no  cost  in  order  to  encourage  broad 
adoption. 

•  Engage  Community 

•  Productize 

Table  5.4-3.  BKCASE  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  The  wiki  format  for  the  SEBoK  has  proven  very  practical,  allowing  for  easy  access  across 
the  worldwide  systems  community  and  has  been  updated  twice  annually  for  three  years 
since  its  first  production  release,  keeping  it  current  and  expanding  its  coverage  of  SE. 

Progress 

(approval, 

adoption) 

•  Almost  20,000  people  visit  the  SEBoK  wiki  every  month.  The  IEEE  Computer  Society  plans 
to  revamp  their  Software  Engineering  Body  of  Knowledge  to  be  structured  as  a  wiki  relying 

on  much  of  the  same  infrastructure  as  was  used  for  the  SEBoK.  The  SEBoK  has  become  an 
acknowledged  strategic  asset  of  INCOSE.  Universities  in  Europe,  Australia,  and  the  US 
have  explicitly  acknowledged  using  GRCSE  to  update  their  SE  graduate  programs. 

5.4. 3. 2  Helix  Project 

Helix  began  in  October  2012  to  examine  the  "DNA"  of  the  systems  engineering  workforce  in  both  DoD 
and  the  defense  industrial  base.  The  project  is  answering  three  questions: 

•  What  are  the  characteristics  of  systems  engineers? 

•  What  enables  them  to  be  effective  and  why? 

•  What  are  employers  doing  to  improve  the  effectiveness  of  their  systems  engineers? 

Based  on  interviews  with  nearly  300  systems  engineers  and  those  who  work  with  systems  engineers,  Helix 
developed  Atlas,  a  theory  of  what  enables  systems  engineers  to  be  effective.  Atlas  describes  the  key 
proficiencies  that  impact  the  effectiveness  of  systems  engineers,  the  several  forces  that  impact  the  level 
of  proficiency  that  systems  engineers  obtain,  how  the  career  paths  of  systems  engineers  progress,  how 
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personal  and  organizational  characteristics  affect  the  evolution  of  systems  engineers,  and  also  provides 
demographic  data  about  systems  engineers,  such  as  their  typical  education  and  how  that  demographic 
data  has  changed  over  time.  The  project  has  expanded  beyond  systems  engineers  in  the  defense 
community  and  is  now  looking  more  broadly  at  systems  engineers  in  such  commercial  sectors  as 
healthcare  and  information  technology. 

Table  5.4-4  shows  the  focus  and  deliverables  in  the  Helix  Project  through  2018. 


Table  5.4-4.  Helix  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Establish  methodology,  small-scale  data 
collection,  publish  early  findings 

Early  findings  report  from  both  DoD  and  the 
defense  industrial  base 

2014 

Large-scale  data  collection  and  analysis;  begin 
regular  publication  of  findings 

Regular  reports  of  findings;  first  version  of  Atlas 

Data  collection  from  broad  commercial 
community  and  understanding  of  what 
enables  other  engineers  to  be  effective  when 
performing  systems  activities 

Regular  reports  of  findings;  mature  version  of 

2015  -  2018 

Atlas;  book  on  what  enables  systems  engineers 
to  be  effective;  user  communities 

The  Helix  transition  action  plan  and  characterization  are  shown  in  Tables  5.4-5  and  5.4-6  below. 


Table  5.4-5.  Helix  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

SERC  formed  an  alliance  with  INCOSE  to  collaborate  broadly  on  research  and 
then  developed  a  specific  agreement  to  collaborate  on  Helix;  that  agreement 
led  to  participation  from  INCOSE's  corporate  members  who  provide  data,  pilot 
Atlas,  and  aid  its  adoption. 

•  Plan  Early 

•  Engage  Community 

•  Support  Centrally 

2 

Helix  has  an  advisory  group  from  the  community  and  periodically  holds 
workshops  with  that  group  and  with  others  to  review  status,  plans,  and  interim 
results. 

•  Engage  Community 

3 

Atlas  is  being  created,  released,  and  piloted  incrementally  in  three  major 
releases  which  are  reviewed  in  workshops  and  other  public  forums,  presented 
at  conferences,  and  described  in  peer-reviewed  papers. 

•  Pilot  Continuously 

•  Balance  Long  & 

Short  Term 

•  Engage  Community 

4 

Helix  is  gradually  creating  and  validating  artifacts  to  help  organizations 
successfully  apply  Atlas  to  aid  their  workforce  development. 

•  Productize 
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Table  5.4-6.  Helix  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Activities  leading  up  to  Atlas  0.5  (the  latest  release)  were  sufficiently  useful  and  practical 
that  several  participating  organizations  have  altered  their  workforce  development 
approaches  to  accommodate  Helix  insights. 

Progress 

(approval, 

adoption) 

•  Helix  has  published  4  technical  reports,  2  conference  papers,  held  two  workshops,  and 
submitted  4  journal  articles. 

•  Organizations  beyond  the  defense  community  are  participating,  expanding  Helix 
outreach. 

5. 4. 3. 3  SEEK  Project 

This  project  addresses  a  gap  in  the  SE  research  literature:  the  lack  of  detailed  case  studies  about  SE 
successes  and  failures.  The  research  developed  its  first  case  studies  in  2014  and  2015,  tailored  to  defense 
education  needs.  These  case  studies  will  support  instruction  at  DAU  and  at  the  Naval  Postgraduate  School, 
the  federal  service  academies,  and  other  government  education  and  training  providers. 

Much  of  SE  and  technical  role  documentation  provides  a  description  of  the  "who",  "what"  and  "when". 
However,  there  is  very  little  guidance  on  the  "how"  for  these  activities.  Case  studies  are  a  tool  that  can  be 
used  to  provide  realism  and  bring  the  systems  engineering  practice  to  the  classroom.  They  can  be  a 
valuable  source  of  lessons  learned  and  underscore  the  effect  of  decision  making.  The  intent  of  this  work 
is  to  provide  the  means  to  support  the  teaching  of  the  "how"  through  the  use  of  case  studies  and  program 
artifacts.  Rather  than  adapting  a  case  study  to  a  course  curriculum,  these  cases  have  been  selected  and 
developed  with  the  end  result  in  mind,  namely  supporting  the  new  DAU  Reliability  &  Maintenance  course. 
The  new  course  consists  of  five  topics  covering  a  significant  (e.g.  Acquisition  Category  1  and  2)  Defense 
Acquisition  program  lifecycle.  The  approach  is  to  create  a  set  of  two  companion  case  studies  that  cover 
the  entire  program  lifecycle,  particularly  through  Test  and  Evaluation.  To  provide  contrast,  one  of  the 
case  studies  will  describe  a  relatively  successful  program,  while  the  other  will  cover  one  that  was  not  as 
successful.  In  addition,  these  cases  and  materials  will  be  assembled  in  a  format  that  allow  broad 
application  into  other  Systems  Engineering  roles  and  activities. 

Obtaining  the  desired  program  information  has  been  a  challenge.  Program  execution  in  years  2016  is 
contingent  upon  acquiring  adequate  program  data  or  will  necessitate  a  shift  to  an  interview  based 
approach.  At  this  point,  no  activities  are  planned  in  2016  pending  confirmation  that  the  necessary 
program  data  for  an  in-depth  case  study  will  be  available. 

Table  5.4-7  shows  the  focus  and  deliverables  in  the  SEEK  Project  through  2018. 
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Table  5.4-7.  SEEK  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Form  project  and  set  up  infrastructure  to  capture 
expert  knowledge 

None 

2014 

Determine  areas  of  interest,  select  target  programs, 
acquire  program  data,  develop  case  study  and 
educational  materials 

Two  case  studies  and  supporting 

educational  materials. 

2015  -  2018 

Refine  and  repeat  process,  increasing  depth  of  current 
cases  or  in  new  programs 

Two  case  studies  and  supporting 
educational  materials  annually  is 
desirable. 

The  SEEK  transition  action  plan  and  characterization  are  show  in  Tables  5.4-8  and  5.4-9  below. 


Table  5.4-8.  SEEK  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Engaged  with  DAU  at  onset  of  project  to  specify  research  outcome 
requirements  so  that  it  may  be  integrated  into  a  DAU  course  being  developed. 

•  Plan  Early 

•  Long  and  Short 

Term 

2 

Created  informal  board  of  advisors  to  review  SEEK  progress  and  provide 
directive  feedback. 

•  Engage  Community 

3 

Presented  case  study  approach  and  draft  of  final  report  and  deliverables  to  the 
DAU  and  Sponsors  during  research  process. 

•  Pilot  Continuously 

Table  5.4-9.  SEEK  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  DAU  has  approved  the  conceptual  plan  for  integrating  SEEK  results  into  their  newly 
being  developed  Reliability  and  Maintainability  course. 

Progress 

(approval, 

adoption) 

•  Early  SEEK  deliverables  will  be  provided  to  the  DAU  and  sponsor  for  feedback 

5.4.4  Experience  Acceleration  Program 

This  research  program  primarily  implements  HCD  Strategy  2  -  Educate  and  Train  Faster.  It  will  include 
projects  aimed  at  creating  automated  learning  environments  that  simulate  real  world  experiences  of 
systems  engineers.  Those  experiences  will  be  vivid  and  realistic  enough  to  significantly  accelerate  the 
learning  and  maturation  of  those  systems  engineers.  One  project  will  evolve  the  current  simulation 
platform,  making  it  ever  more  robust  and  capable  and  enabling  quicker  and  easier  construction  of  new 
experiences.  Other  projects  will  add  to  the  current  catalog  of  experiences,  developing  new  experiences 
that  use  the  simulation  platform.  Experiences  will  vary  based  on  the  size  and  types  of  systems  being 
acquired,  the  acquisition  lifecycle,  the  novelty  of  the  technology  being  acquired,  and  other  parameters  of 
interest.  Over  the  five-year  period  from  2014-2018,  other  organizations  will  join  the  SERC  in  improving 
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the  experience  platform  and  in  developing  additional  experiences,  creating  a  marketplace  for  experience 
acceleration.  Table  5.4-10  offers  a  description  of  these  projects  and  which  strategies  they  primarily 
support.  Table  5.4-11  shows  the  focus  and  deliverables  in  the  Experience  Accelerator  Project  through 
2018. 

This  program  is  unusual  in  that  the  primary  project,  Experience  Accelerator,  has  had  relatively  little  Core 
funding.  Moreover,  relatively  little  Core  funding  is  expected  in  the  future.  DAU  has  been  and  remains  the 
primary  sponsor  for  the  effort.  Other  DoD  organizations  are  expected  to  provide  additional  non-Core 
funding  to  create  new  Virtual  Experiences  beginning  in  2016. 


Table  5.4-10.  Projects  in  the  Experience  Acceleration  Program 


Projects 

Started 

Purpose 

Primary  HCD 
Supported 
Strategies 

Experience 

Accelerator 

2010 

Create  the  "engine"  that  will  be  used  to  host  a  wide  range  of 
experiences,  develop  the  first  virtual  experiences  that  use  the 
engine,  and  validate  the  experience  accelerator  concept  through 
trial  use.  Keep  improving  the  engine  over  time  as  a  broader  set  of 
experiences  are  created  and  trialed  with  ever  more  students. 

Create  an  open  vibrant  community  that  will  develop  additional 
virtual  experiences  that  can  be  shared  within  the  defense  industrial 
base  and  DoD. 

2 

Additional 

Expected 

Relying  on  non-Core  investment,  develop  an  increasingly  broader 

Virtual 

2016  and 

and  richer  set  of  virtual  experiences  that  are  hosted  on  the 

2 

Experiences 

beyond 

Experience  Accelerator  engine 

Table  5.4-11.  Experience  Accelerator  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Build  experience  accelerator  engine,  first 
experiences,  and  validate  concept  with  first 

students 

Increasingly  sophisticated  versions  of  Experience 
Accelerator  engine,  first  virtual  experience,  first 
use  by  students 

2014 

Create  an  increasingly  more  complete  and 
sophisticated  engine,  and  tools  to  build 
virtual  experiences 

New  versions  of  increasingly  more  capable  engine 
with  first  set  of  prototype  tools  to  create 
experiences 

2015 

Begin  to  transition  EA  into  use  at  the  DAU 

Early  deployment  at  DAU,  increasingly  more 
capable  engine  and  tools,  and  learning 

assessments 

2016-2018 

Develop  tools  that  enable  a  vibrant  EA 
community  for  experience  development 
and  learning  assessment 

New  experiences,  increasingly  more  capable 
engine  and  tools,  learning  evaluations, 
deployment  in  marketplace  community  with 
suitable  governance  structure 

The  Experience  Accelerator  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.4-12 
and  5.4-13  below. 
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Table  5.4-12.  Experience  Accelerator  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

Worked  closely  with  DAU  to  determine  project  concept  and  scope 

•  Plan  Early 

2 

Delivered  a  number  of  Pilots  for  instructors  and  students,  updating  the 
Experience  Accelerator  based  on  detailed  feedback 

•  Long  and  Short 

Term 

•  Pilot  Continuously 

3 

Created  Experience  Accelerator  User  Group  to  publicize  results  and  provide 
feedback  on  development.  Hold  workshops  at  NDIA  SE  and  INCOSE 
International  Symposium  conferences 

•  Engage  Community 

4 

Develop  tools  to  improve  the  capabilities  of  users  to  customize  experiences 
for  their  use 

•  Productize 

Table  5.4-13.  Experience  Accelerator  Project  Transition  Plan 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  DAU  desires  the  integration  of  the  Experience  Accelerator  into  their  curriculum.  The 
Experience  Accelerator  User  Group  is  composed  of  16  members  from  multiple 
organizations  in  the  Federal  Government,  FFRDCs,  industry  and  academia. 

Progress 

(approval, 

adoption) 

•  The  Experience  Accelerator  has  undergone  several  instructor  and  student  pilots  which 
resulted  in  140+  suggestions  for  improvement.  Many  of  these  have  been  incorporated 
into  the  Experience  Accelerator  and  are  now  ready  for  use.  The  first  set  of  tools  have 
been  developed  and  evaluated  by  external  researchers.  A  workshop  demonstrating  these 
tools  was  conducted  at  INCOSE  International  Symposium  2015. 

5.4.5  Systems  Engineering  and  Technical  Leadership  Education  Program 

This  research  program  primarily  implements  HCD  strategies  2  and  3  -  Educate  and  Train  Faster  and 
Develop  Effective  Technical  Leaders.  It  currently  includes  two  primary  projects:  the  Engineering  Capstone 
Marketplace  Project  and  the  Technical  Leadership  Project.  The  Capstone  Project  has  mixed  Core  and  US 
Special  Operations  Command  funding.  Even  more  extreme  than  the  Experience  Acceleration  Program  just 
described,  all  of  the  investment  for  the  Technical  Leadership  Project  has  come  from  DAU  -  none  has  come 
from  Core  funding.  Nevertheless,  it  is  included  here  because  it  has  been  so  critical  to  the  HCD  research 
effort. 

Table  5.4-14  offers  a  description  of  these  projects  and  which  strategies  they  primarily  support. 
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Table  5.4-14.  Projects  in  the  Systems  Engineering  and  Technical  Leadership  Education  Program 


Projects 

Started 

Purpose 

Primary  HCD 
Supported 
Strategies 

Engineering 

Capstone 

Marketplace 

2010 

Originally  intended  to  show  how  to  conduct  multidisciplinary  senior 
capstone  projects  in  classical  engineering  programs,  especially  those 
that  increase  awareness  of  and  appreciation  for  DoD  applications. 
Building  on  its  early  success,  the  project  has  morphed  into  a 
marketplace  where  companies  and  government  organizations  can 
post  on  a  website  problems  suitable  for  senior  capstone  projects. 
Students  from  multiple  universities  form  teams  to  work  on  those 
projects  under  the  supervision  of  faculty  and  the  posting 
organization.  Although  small  today,  this  marketplace  model  has  the 
potential  to  scale  nationwide,  involving  thousands  of  students  in 
hundreds  of  projects  and  universities. 

2,4 

Technical 

Leadership 

2010 

Develop  innovative  ways  to  teach  technical  leadership  to  the  DoD 
acquisition  workforce,  including  not  only  systems  engineers,  but 
also  others  who  must  understand  technical  leadership,  such  as 
program  managers.  Iteratively  pilot  the  resulting  courses  and 
integrate  them  into  the  DAU  curriculum. 

2,3,4 

5.4. 5.1  Engineering  Capstone  Marketplace  Project 

The  Engineering  Capstone  Marketplace  (ECM)  Project  is  the  evolution  of  research  begun  in  2010,  which 
showed  that  a  multidisciplinary  senior  capstone  project  could  enhance  development  of  systems  SE 
competencies  and  increase  interest  in  SE.  ECM  is  now  in  its  third  year  of  matching  engineering  students 
working  on  their  capstone  or  senior  design  with  sponsors  who  provide  challenging  real  world  problems 
and  dedicated  mentors  or  subject  matter  experts.  This  matching  is  done  through  the  ECM  website20. 
Current  efforts  include  development  of  the  process  and  infrastructure  to  affordably  scale  this  approach 
nationwide  and  improve  how  thousands  of  students  are  taught  engineering  across  the  US. 

Sponsoring  organizations  from  within  the  DoD  have  provided  significant  and  challenging  projects  that 
have  engaged  engineering  students  from  multiple  engineering  schools.  The  project  sponsors  provide 
funding  for  research,  prototyping,  testing,  travel,  etc.,  and  a  dedicated  subject  matter  expert(s)  to  work 
with  individual  teams.  These  Capstone  Marketplace  projects  provide  an  important  opportunity  for 
participating  students  to  enhance  their  SE  knowledge  and  experience  by: 

•  Working  on  real  and  complex  problems  with  engaged  mentors 

•  Working  on  projects  that  require  a  multi-disciplinary  team/approach 

•  Implementing  and  gaining  experience  with  systems  engineering  approaches  and  processes. 

Project  sponsors  have  received  many  useful  results.  Several  projects  have  led  to  follow  on  efforts  and 
discussions  have  been  initiated  with  the  sponsor  of  two  of  these  projects  to  develop  field  deployable 
devices.  These  future  engineers  are  entering  the  workforce  with  enhanced  SE  skills  and  understanding 


20  See  http://www.capstonemarketplace.org/. 
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that  can  help  address  the  critical  challenge  of  developing  the  next  generation  of  SE  talent  for  future  DoD 
and  defense  industry  needs. 

Table  5.4-15  shows  the  focus  and  deliverables  in  the  ECM  Project  through  2018. 


Table  5.4-15.  Engineering  Capstone  Marketplace  Project  Timeline 


Year 

Focus 

Key  Deliverables 

Pre  -  Core 

Develop  and  validate  on  a  small  scale 
approaches  to  fostering  multidisciplinary 
senior  capstone  projects  both  within  single 
universities  and  across  universities,  Research 
and  develop  a  means  to  broaden 
participation  in  SE  related  Capstone  Projects 

Reports  on  how  to  foster  multidisciplinary  senior 
capstone  projects,  experience  reports  from 
participating  universities,  infrastructure  to 
support  nationwide  expansion  of  the  approach.  A 
pilot  web  based  marketplace  for  capstone 
engineering  projects.  Assessment  of  successful 
practices  for  managing  multi-university,  multi¬ 
disciplinary  sponsored  projects. 

2014 

Enhance  and  improve  the  ECM  website  to 
allow  for  broader  participation  and  scale-up 

Developed  a  more  robust  and  scalable  website  for 
sponsors,  faculty  and  students. 

Broadened  sponsor  and  university  participation, 
developed  resources  and  project  guidelines 

2015 

Redevelop  the  ECM  website  to  enable  project 

coordination  and  assessment  tools.  Increase 
outreach  and  communication  with  early 
adaptors  to  increase  participation  in  the 
marketplace 

New  Capstone  Marketplace  website.  Developed 
resources  and  documentation  to  disseminate 

information  on  ECM  to  government,  industry,  and 
academia.  Develop  SE  learning  assessment 
processes.  Develop  informational  and/or 
instructional  resources  for  implementation  of  SE 
into  capstone  projects. 

2016- 

2018 

Expand  the  Marketplace  to  draw  hundreds  of 
students  from  all  across  the  country  and 
across  many  universities,  industrial  and 
government  organizations  to  sponsor 
multidisciplinary  capstone  problems 

Robust  infrastructure  capable  of  supporting  large- 
scale  involvement  of  universities,  students,  and 
organizations,  experience  reports,  outreach 
mechanisms  to  engage  a  nationwide  audience  of 
students,  universities,  and  organizations,  analysis 
of  the  value  and  impact  of  the  Marketplace 

The  ECM  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.4-16  and  5.4-17. 
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Table  5.4-16.  Engineering  Capstone  Marketplace  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

1 

The  transition  of  the  ECM  from  an  internal  SERC  capability  to  a  stand-alone 
website  that  can  be  readily  transitioned  was  part  of  the  project  goals  which 
were  refined  in  2014.  Steps  that  have  been  taken  in  the  regard  include; 
registration  of  a  non-SERC  domain  name  www.capstonemarketplace.org, 
outreach  to  the  academic  capstone  design  community,  website  redesign  to 
reduce  administrative  load,  development  of  web-site  resources  that  better 
enable  users  to  manage,  plan,  and  conduct  their  projects. 

•  Plan  Early 

•  Long  and  Short  Term 

•  Productize 

2 

Outreach  to  both  of  the  key  national  capstone  project  user  groups  have  been 
conducted  with  participation  and  distribution  of  ECM  literature  at  NDIA 
events,  publication  and  presentation  of  a  ECM  paper  and  distribution  of  ECM 
literature  at  the  2015  American  Society  for  Engineering  Education  (ASEE) 
conference.  Various  other  efforts  to  engage  government  sponsors  for  ECM 
projects  continues.  Development  of  ECM  project  briefing  materials  is  ongoing. 

•  Engage  Community 

3 

Results  of  recently  completed  ECM  student  projects  along  are  being  assessed. 
Outreach  to  faculty  advisors  and  sponsor  liaisons  to  review  results  and 
determine  lessons  learned  are  ongoing. 

•  Pilot  Continuously 

4 

A  plan  to  develop  online  tools  for  the  ECM  website  that  will  better  enable 
dissemination  of  basic  SE  tools  and  processes  and  assessment  of  SE  learning  is 
being  developed. 

•  Long  and  Short  Term 

•  Engage  Community 

Table  5.4-17.  Engineering  Capstone  Marketplace  Project  Transition  Characteristics 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  In  the  2014-15  academic  year,  9  of  10  student  projects  were  successfully  conducted  by  4 
academic  institutions  and  8  new  student  projects  are  underway  in  the  2015-2016 
academic  year.  Both  the  sponsors  and  the  faculty  advisors  have  expressed  interest  or 
committed  to  continued  participation. 

Progress 

(approval, 

adoption) 

•  2  conference  papers  have  been  published  including  a  recent  publication  and  presentation 
to  the  ASEE  Design  in  Engineering  Education  Division.  It  is  this  group  that  is  the  most  likely 
to  adopt  the  ECM  as  a  resource  for  faculty  and  students.  This  group  has  agreed  to  link  the 
ECM  to  their  Capstone  Design  Hub  website  and  disseminate  information  on  the  ECM  to  its 
members. 

•  The  very  successful  participation  of  the  Special  Operations  Forces  (SOF)  Acquisition, 
Technology  &  Logistics  Science  and  Technology  Office  in  the  Capstone  Marketplace  was  a 
prominent  feature  of  3  SOF/Industry  roundtable  discussions  at  the  recent  NDIA  SOF 

Industry  Conference. 

5.4. 5. 2  Technical  Leadership  Project 

The  Technical  Leadership  Project  began  in  2010  to  evaluate  the  hypothesis  that  the  technical  leadership 
capabilities  of  high  potential,  senior  DoD  systems  engineers  and  technologists  could  be  accelerated 
through  an  educational  program  in  technical  leadership.  This  research  has  resulted  in  the  creation  of  an 
innovative  approach  to  educating  technical  leaders  through  three  lenses:  systems,  business,  and 
enterprise.  A  series  of  three  five-day  courses  have  been  prototyped,  piloted  and  are  in  the  process  of 
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being  transitioned  to  the  DAU.  Each  course  contains  a  series  of  independent  readings,  lectures,  case 
studies,  and  student  in-class  exercises  to  accelerate  systems  technical  leadership  learning.  The  courses 
take  the  student  from  (a)  leading  systems  development  in  the  face  of  uncertainty  and  ambiguity  to  (b) 
understanding  how  commercial  businesses  or  organizations  accountable  for  multi-system  and  multi¬ 
customers  strategize,  operate  and  measure  performance  to  (c)  the  technical  leadership  expectations  of 
an  enterprise  senior  technical  leader  responsible  for  assessing  and  adapting  multi-nodal  structural  and 
activity-based  processes  within  DoD  or  commercial  enterprises. 

Ongoing  research  supports  the  transition  to  the  DAU  of  the  three  SYS  350  series  Systems  Engineering 
Technical  Leadership  prototype  courses  to  address  the  need  for  technical  leadership  education  in  parallel 
with  functional  training.  The  transition  approach  includes  (a)  assignment  of  a  DAU  SYS  350  faculty 
instructional  team;  (b)  SERC  review,  update,  and  refinement  of  the  current  prototype  SYS  350A  Systems, 
SYS  350B  Business,  and  SYS  350C  Enterprise  technical  leadership  courses;  and  (c)  two  transition  pilots  for 
each  of  the  three  350  series  courses.  Further,  for  each  of  the  350  A/B/C  course  pilots,  the  first  of  each 
A/B/C  transition  pilot  will  be  led  by  the  SERC  research  team  and  supported  by  the  DAU  Faculty  team.  The 
second  transition  pilot  for  each  of  the  A/B/C  transition  pilots  will  be  then  be  led  by  the  DAU  faculty  team 
and  supported  by  the  SERC  team  as  required. 

Table  5.4-18  shows  the  focus,  deliverables,  and  investment  in  the  Technical  Leadership  Project  through 
2017,  at  which  time  it  is  likely  to  conclude. 


Table  5.4-18.  Technical  Leadership  Education  Projects  Timeline 


Year 

Focus 

Key  Deliverables 

Core  and 

Previous 

Investment 

Pre  -  Core 

Create  and  pilot  three  courses  for 
the  systems,  business,  and 
enterprise  lenses 

Course  materials  and  reports  from  pilot 
classes 

none 

2014- 

2017 

Refine  the  courses  and  integrate 
them  into  the  DAU  curriculum 

Final  versions  of  course  materials  for  DAU- 
led  pilots  and  for  integration  into  the  DAU 
curriculum;  reports  from  pilot  classes 

none 

The  Technical  Leadership  Project  transition  action  plan  and  characterization  are  shown  in  Tables  5.4-19 
and  5.4-20  below. 


Table  5.4-19.  Technical  Leadership  Project  Transition  Actions 


# 

Transition  Action 

Principles  Implemented 

Engaged  with  DAU  at  onset  of  project  to  specify  research  outcome 

•  Plan  Early 

i 

requirements  so  that  the  research  results  can  be  deployed  at  DAU.  Developed 

•  Long  and  Short 

transition  plan  with  sponsors. 

Term 

Worked  closely  with  DAU  to  develop  relationship  with  Missile  Defense  Agency 

2 

to  provide  an  additional  set  of  pilots  and  sites  for  broader  adoption.  Formed 
transition  teams  with  sponsor. 

•  Engage  Community 
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# 

Transition  Action 

Principles  Implemented 

3 

Developed  and  executing  a  continuous  piloting  process.  Conducted  nine  pilots 
with  feedback  generated  from  over  5,200  faculty-student  pilot  contact  hours. 

•  Pilot  Continuously 

4 

Over  100  lecture,  case  study,  exercise,  simulations,  and  group  project 
classroom  segments  were  developed  and  tested. 

•  Productize 

Table  5.4-20.  Technical  Leadership  Project  Transition  Characterization 


Characteristic 

Evidence 

Readiness 

(relevance, 

practicality) 

•  Both  DAU  and  the  Missile  Defense  Agency  have  piloted  these  courses  and  are  seeking 
their  transition  from  research  into  practice.  The  current  research  is  focused  on  bringing 
about  this  transition. 

Progress 

(approval, 

adoption) 

•  Conducted  nine  pilots  with  feedback  generated  from  over  5,200  faculty-student  pilot 
contact  hours.  The  research  task  for  2016-2017  will  result  in  these  courses  being  taught 
by  DoD  instructors  and  formally  integrated  into  DAU  and  the  Missile  Defense  Agency. 

5.4.6  HCD  Area  Non-Core  Funded  Projects 

During  the  time  of  the  Technical  Plan,  the  SERC  has  executed  four  non-Core-funded  HCD  Area  projects  as 
briefly  described  in  Table  5.4-21  below.  Additionally,  one  Core-funded  project,  Helix,  received 
complementary  funding  from  the  Intelligence  Community,  and  one  project,  Experience  Accelerator, 
augmented  its  primary  DAU  funding  with  Core  funds  to  improve  automation  to  more  quickly  create  new 
virtual  experiences. 


Table  5.4-21.  HCD  Area  Non-Core  Funded  Projects 


Project 

Sponsor 

Description 

Science  &  Technology 
Portfolio  Development 

DAU 

Develop,  synthesize,  and  validate  principles  for  formulating  and 
executing  a  strategic  plan  for  Science  &  Technology  development, 
with  the  goal  of  enhancing  the  capability  of  the  DoD's  acquisition 
workforce. 

Army  Career  Development 

Model 

Army 

Expand  an  earlier  model  for  developing  Army  civilian  systems 
engineers  in  the  Army's  Engineering  Acquisition  Career  Field  to 
assist  in  developing  Lead  and  Chief  Systems  Engineers  for  Key 

Leader  Positions. 

Helix 

1C 

Expand  the  scope  of  research  to  understand  differences  between 
systems  engineers  in  the  defense  community  and  those  in 
commercial  organizations  and  understand  on  which  aspects  of 
systems  engineering  should  classic  engineers  be  effective. 

Experience  Accelerator 

DAU 

Develop  a  capability  to  use  simulation  that  is  visceral  enough  to 
put  the  learner  in  an  experiential,  emotional  state  and  effectively 
compress  time  and  greatly  accelerate  the  learning  of  a  systems 
engineer  faster  than  would  occur  naturally  on  the  job. 
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Project 

Sponsor 

Description 

Advanced  Technical 
Leadership 

DAU 

Research,  synthesize,  and  validate  curriculum  content  and 
structure  for  future  DoD  systems  engineering  and  technical 
leaders  at  the  senior  and  executive  levels. 

Leadership  Development 
Framework  for  the 

Technical  Acquisition 

Workforce 

DAU 

Define  a  balanced  career  path  structure  for  the  DoD  technical 
acquisition  workforce  that  combines  training,  education, 
experiential  learning,  developmental  assignments,  and  other 
elements  leading  to  appropriate  knowledge,  skills  and  abilities 
suitable  at  each  stage  of  a  technical  career. 

5.5  Infrastructure  Development 

The  SERC  Vision  states,  in  part,  that  at  the  end  of  this  5-Year  plan: 

•  The  SERC  operates  the  world's  largest  and  most-visited  SE Research  Website,  including  the  largest 
and  best-organized  SE  Research  experience  base. 

•  It  continues  to  provide  leadership  in  evolving  the  SE  Body  of  Knowledge. 

•  It  runs  the  most  widely  attended  and  highest-rated  SE  Webcast  series. 

The  SERC  will  develop  the  infrastructure  that  can  be  used  by  all  of  the  research  programs  to  support 
achieving  these  elements  of  the  Vision.  The  strategy  is  to  focus  on  two  specific  areas:  the  SERC  Website 
and  the  SERC  Innovation  and  Demonstration  Lab.  This  work  is  scoped  to  be  accomplished  through  a  mix 
of  SERC  investment  funds  and  Core  funds. 


5.5.1  SERC  Website 

The  SERC  website  is  intended  to  become  the  world's  premier  SE  research  website  and  researcher 
collaboration  portal.  As  stated  in  the  SERC  vision,  it  is  intended  to  become  "the  world's  largest  and  most- 
visited  SE  Research  website,  including  the  largest  and  best-organized  SE  Research  experience  base".  To 
support  this  objective,  the  SERC  website  was  redesigned  in  2014  and  has  regular  changes,  additions  and 
improvements.  Through  a  simple,  clean,  and  intuitive  web  interface,  site  visitors  can: 

•  Learn  about  the  SERC; 

•  View  news  and  events,  not  just  SERC-related  but  those  relevant  to  the  entire  SE  community; 

•  Review  and  download  current  and  historical  SERC  publications,  including  annual  reports,  technical 
plans,  research  papers,  technical  reports,  and  associated  articles; 

•  View  presentations  from  past  SERC  events,  including  SERC  Sponsor  Research  Reviews  and 
Doctoral  Students  Forums; 

•  Gain  an  understanding  of  SERC-affiliated  programs,  such  as  the  SERC  Doctoral  Fellows  Program; 
and 

•  With  appropriate  credentials,  log  in  to  the  SERC  Collaboration  Portal,  File  Storage,  and  Product 
Distribution  environment. 

All  of  this  functionality  exists  on  the  current  SERC  website  with  the  exception  of  the  collaboration  portal, 
file  storage,  and  product  distribution  environment.  The  remaining  functions  will  be  implemented  in 
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concert  with  the  development  of  the  SIDL  described  in  the  next  section.  Initial  operational  capability  of 
the  portal  is  anticipated  in  2016. 


5.5.2  SERC  Innovation  and  Demonstration  Lab  (SIDL) 

As  mentioned  earlier,  one  of  the  SERC's  major  goals  for  2018  is  to  operate  the  world's  largest  and  most- 
visited  SE  Research  Web  site,  including  the  largest  and  best-organized  SE  Research  experience  base. 
Rather  than  strictly  share  written  materials,  the  SERC  will  share  its  MPTs  with  the  systems  research 
community.  The  SIDL  and  the  SERC  Website  will  be  the  primary  vehicles  for  these  collaboration  and 
sharing  efforts. 

The  SIDL  will  be  the  first  stop  for  an  SE  researcher.  The  virtual  face  of  the  facility  will  be  integrated  within 
the  SERC  website,  effectively  acting  as  a  repository  for  the  source  code,  executable  files,  data  files, 
technical  reports,  white  papers,  and  documentation  developed  and  matured  as  part  of  SERC  research.  In 
addition,  the  site  will  contain  a  description  of  MPTs  along  with  online  demonstrations,  videos,  instructions, 
and  downloads  for  SERC  developed  tools,  open  source  tools,  and  URLs  of  often-used  commercial  tools. 
This  material  will  be  appropriately  linked  to  the  reports,  papers,  and  other  documentation  relating  to  SERC 
sponsored  research.  At  a  future  date,  this  may  be  linked  to  non-SERC  sponsored  SE  research  as  well. 
Wikis  and  chat  rooms  will  provide  the  means  by  which  SE  researchers  can  share  their  experiences  with 
these  tool  sets.  Another  possibility  for  a  SERC  hosted  and  distributed  tool  is  a  Linux  distribution  packaged 
with  all  available  open  source  SE  tools  that  can  run  in  this  environment.  This  software  could  be  made 
available  as  a  downloadable  "SE  research  starter  kit."  The  SERC  would  provide  expertise  and  assistance 
to  SERC  collaborating  researchers  to  ensure  that  their  research  tools,  data,  and  results  are  shareable  on 
the  SIDL. 

The  central  physical  facility  is  currently  at  Stevens  Institute  of  Technology  as  an  immersive  environment 
in  which  SERC  researchers  can  dynamically  visualize  their  data,  test  hypotheses,  develop  appropriate 
algorithms  and  associated  MPTs,  and  draw  conclusions.  The  SIDL  provides  a  means  to  "package"  and 
"serve"  demonstrations  on  demand,  meaning  that  all  configurations  and  settings  are  stored  within  the 
demonstration  "package."  So,  when  one  demonstration  requires  specific  settings  in  the  Lab,  it  will  not 
conflict  with  another  demonstration's  settings  or  lab  configuration  requirements.  The  facility  consists  of 
multiple  vertically-mounted  touchscreen  displays  along  with  the  necessary  computer  hardware  to  run 
simulations  and  drive  the  graphical  environment.  Beyond  the  hardware  and  software,  this  lab  is  designed 
to  provide  multiple  benefits  to  the  SERC  researchers  and  staff.  The  SIDL  also  provides  a  means  for  SERC 
researchers  to  collaborate  remotely  and  "come  together"  in  a  lab  environment.  Additional  physical 
facilities  of  various  capabilities  will  be  supported  at  other  SERC  collaborating  institutions.  These  facilities 
will  be  supported  by  their  host  collaborating  institutions  but  will  provide  an  environment  to  demonstrate 
SERC  research  to  interested  parties  near  those  sites. 

The  initial  goal  and  pilot  case  of  the  SIDL,  completed  in  July  2015,  was  to  visually  demonstrate  three 
separate  research  projects  and  show  how  each  of  these  research  projects  could  be  used  collaboratively 
to  produce  results,  which  could  not  be  achieved  by  the  projects  separately.  While  each  of  these  research 
tasks  provides  significant  capabilities  and  reveals  valuable  results  on  its  own,  the  integration  of  the  MPTs 
provides  even  greater  capabilities  and  insights,  and  ultimately  provides  more  valuable  results  to  the 
SERC's  sponsors. 
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6  New  Project  Initiation 


There  are  numerous  sources  for  new  SERC  projects.  In  many  instances,  the  government  sponsors  have 
emerging  critical  problems  that  require  the  insights  of  research  efforts.  In  these  cases,  an  open  call  may 
be  made  to  solicit  proposals  from  within  the  SERC  collaborator  base  to  find  the  best  research  ideas.  These 
are  often  reviewed  by  SERC  leadership,  the  SERC  Research  Council,  and  the  government  sponsors.  In 
other  cases,  the  sponsor  may  have  already  identified  leading  researchers  for  the  efforts  who  write  a 
proposal  for  the  work.  In  both  approaches,  the  SERC  strives  to  form  collaborative  teams  when  it  is 
appropriate.  The  research  community  may  develop  research  concepts  that  are  presented  to  various 
federal  sponsoring  agencies,  some  of  which  may  choose  to  sponsor  the  research.  Each  research  program 
also  develops  new  research  topics  that  are  related  to  their  ongoing  research  efforts.  Finally,  the  SERC  has 
initiated  a  formal  process  of  incubation  grants  to  provide  funding  for  SERC  collaborators  so  that  they  can 
develop  their  research  ideas  to  the  point  where  formal  proposals  can  be  made.  Research  ideas  from  the 
incubation  grants  are  described  below. 

As  described  in  this  Technical  Plan,  the  SERC  typically  performs  research  on  20  to  25  active  tasks 
simultaneously,  exploring  well-defined  topics  that  are  aligned  with  the  SERC's  research  strategy.  While  it 
is  believed  that  that  the  aforementioned  research  programs  have  a  great  potential  to  have  a 
transformative  impact  on  the  DoD,  1C,  and  beyond,  there  is  a  need  to  support  new  ideas  in  their  infancy 
that  may  become  critical  research  programs  for  emerging  challenges.  This  "incubation"  capability  will  be 
supported  by  an  annual  open  call  to  the  SERC  research  collaborating  universities  to  propose  early  stage 
research  that  can  be  nurtured  through  relatively  small  levels  of  seed  funding. 

The  initial  open  call  took  place  in  September  2014  with  the  objective  of  identifying  and  developing  several 
short  white  papers  outlining  research  programs  with  a  significant  potential  to  improve  the  practice  of 
engineering  systems.  A  total  of  29  responses  were  received  and  reviewed  with  the  sponsor  and  the 
Research  Council.  Of  these,  the  following  five  proposals  were  selected  and  funded: 

1.  Mark  Avnet,  Texas  A&M,  A  Multilevel  Framework  of  System  Safety:  Technical  Failures,  Fluman 
Factors,  Organizational  Culture,  and  Societal  Influence 

2.  LiGuo  Huang,  SMU,  Detecting  and  Evaluating  Technical  Debt  of  Software  Systems 

3.  Azad  Madni,  USC,  Formal  Methods  in  Resilient  Systems  Design  using  a  Flexible  Contract  Approach 

4.  Kevin  Sullivan,  UVa,  Foundations  of  Systems  Engineering 

5.  Gary  Witus,  Wayne  State,  Policies  and  Practices  for  Model-Centric  Government-Industry 
Collaborative  Environments  for  Systems  Engineering  and  Development 

The  results  of  these  efforts  will  be  considered  for  Core  funding  and  potentially  non-Core  funding  as  well. 
The  SERC  incubator  timeline  is  shown  in  Table  6.1-1. 
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Table  6.1-1.  SERC  Project  Incubation  Timeline 


Year 

Focus 

Key  Deliverables 

Pre-Core 

None  -  new  start  in  2014 

2014 

Create  proposal  format  and  process  in 
consultation  with  SERC  sponsors  with  input  from 
SERC  collaborators 

Announcement  of  incubator  proposal 
solicitation  to  SERC  collaborating  universities 

2015-2018 

Selection  of  top  proposals  for  funding 

Award  of  seed  funding  to  selected  projects 

Contract  Number:  HQ0034-1 3-D-0004 


February 

76 


2016 


SERC  201 4-201 8  Technical  Plan  Update 


UNCLASSIFIED  /APPROVED  FOR  PUBLIC  RELEASE 


Acronyms  and  Abbreviations 

ABET  -  Accreditation  Board  for  Engineering  and  Technology 

AIAA  -  The  American  Institute  of  Aeronautics  and  Astronautics 

ASD(R&E)  -  Assistant  Secretary  of  Defense  for  Research  and  Engineering 

ASEE  -  American  Society  for  Engineering  Education 

AWB  -  Analytic  Workbench 

BBP  -  Better  Buying  Power 

BKCASE  -  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering 

CMU  -  Carnegie  Mellon  University 

CONOPS  or  ConOps  -  Concept  of  Operations 

CRADA  -  Cooperative  Research  and  Development  Agreement 

CSER  -  Conference  on  Systems  Engineering  Research 

DATASEM  -  Demonstration  and  Analysis  Tool  for  Adaptive  Systems  Engineering  Management 

DAU  -  Defense  Acquisition  University 

DoD  -  Department  of  Defense 

DSL  -  Domain  Specific  Language 

EA-  Experience  Accelerator 

ECM  -  Engineering  Capstone  Marketplace 

ERS  -  Engineered  Resilient  Systems 

ESOS  -  Enterprises  and  Systems  of  Systems 

FACT  -  Framework  for  Assessing  Cost  and  Technology 

FFRDC  -  Federally  Funded  Research  and  Development  Center 

FILA-SoS  -  Flexible  and  Intelligent  Learning  Architectures  for  Systems  of  Systems 

GRCSE  -  Graduate  Reference  Curriculum  for  Systems  Engineering 

HCD  -  Fluman  Capital  Development 

1C  -  Intelligence  Community 

IEEE-CS  -  Institute  of  Electrical  and  Electronics  Engineers  Computer  Society 

IMCSE  -  Interactive  Model-Centric  Systems  Engineering 

INCOSE  -  International  Council  on  Systems  Engineering 

JHUAPL-Johns  Hopkins  University  Applied  Physics  Lab 

KSS  -  Kanban  Scheduling  System 

MBSE  -  Model-Based  Systems  Engineering 
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MPT  -  Method,  Process,  and/or  Tool 

MS-SE  -  Masters  Degree  in  Systems  Engineering 

NAVSEA-  Naval  Sea  Systems  Command 

NCSOS  -  Net-Centric  Systems  of  Systems 

NDIA-  National  Defense  Industrial  Association 

NIST  -  National  Institute  of  Standards  and  Technology 

NSWC  -  Naval  Surface  Warfare  Center 

OSD  -  Office  of  the  Secretary  of  Defense 

PhD-SE  -  Doctor  of  Philosophy  Degree  in  Systems  Engineering 

PI  -  Principal  Investigator 

QSAM  -  Quantitative  Schedule  Acceleration  Model 

QTR  -  Quantitative  Technical  Risk 

SE  -  Systems  Engineering 

SE&M  -  Systems  Engineering  and  Management 

SEBoK  -  Systems  Engineering  Body  of  Knowledge 

SEEK  -  Systems  Engineering  Expert  Knowledge 

SEMT-  Systems  Engineering  and  Systems  Management  Transformation 

SERC  -  Systems  Engineering  Research  Center 

SIDL-SERC  Innovation  and  Demonstration  Laboratory 

SOF  -  Special  Operations  Forces 

SoS  -  System  of  Systems 

SoSE  -  System  of  Systems  Engineering 

SQ-  System  Qualities 

STEM  -  Science,  Technology,  Engineering,  and  Math 
SysML  -  Systems  Modeling  Language 

TARDEC  -  Tank  and  Automotive  Research,  Development,  and  Engineering  Center 
TS  -  T rusted  Systems 

UARC  -  University  Affiliated  Research  Center 
USNWR  -  US  News  and  World  Report 

USD(AT&L)  -  Undersecretary  of  Defense  for  Acquisition,  Technology  &  Logistics 
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